Error positioning method and device, equipment and storage medium

Abstract

The embodiment of the invention relates to the technical field of data processing, in particular to an error positioning method and device, equipment and a storage medium, and aims at quickly positioning errors occurring in a large language model. The method comprises the following steps: copying a calculation parameter corresponding to a target operator of each layer of a target model in first calculation equipment and a first result parameter into second calculation equipment; executing the same calculation step as the target operator according to the calculation parameter through a deep learning framework in the second calculation equipment to obtain a second result parameter; and under the condition that an error value between the first result parameter and the second result parameter exceeds a preset first error threshold value, performing intermediate result comparison on calculation steps of the target operator in the first calculation equipment and the second calculation equipment, and determining an error position in the target model.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and more specifically, to an error location method, apparatus, device, and storage medium. Background Art

[0002] During the development of a large language model chip, when debugging and optimizing the large language model, if an error occurs in the calculation of the large language model, it is necessary to locate the location of the error as soon as possible. In the related art, the error of the large language model is located by performing comprehensive functional verification and performance analysis on the design of the large language model in a simulation environment, or by inserting a large number of print statements and log records during the chip design process and collecting runtime information to locate the error of the large language model.

[0003] In the related art, it is impossible to quickly locate the specific location and cause of errors in large language models, and it takes a lot of time and resources to complete the debugging process. Summary of the invention

[0004] The embodiments of the present application provide an error location method, apparatus, device, and storage medium, which are intended to quickly locate errors that occur in a large language model.

[0005] A first aspect of an embodiment of the present application provides an error location method, the method comprising: Copying the calculation parameters and the first result parameters corresponding to the target operator of each layer of the target model in the first computing device to the second computing device; By using the deep learning framework in the second computing device, the same computing steps as those of the target operator are executed according to the computing parameters to obtain a second result parameter; When the error value between the first result parameter and the second result parameter exceeds a preset first error threshold, intermediate results of the calculation steps of the target operator are compared in the first computing device and the second computing device to determine the error position in the target model.

[0006] Optionally, the method further comprises: Performing error type judgment on the error in the target model to determine the error type corresponding to the error; An error report is generated including the error type.

[0007] Optionally, before copying the calculation parameters corresponding to the target operator of each layer of the target model in the first computing device and the first result parameters to the second computing device, the method further includes: According to the structure of the target model, determining the main operators included in each layer of the target model; The main operator is used as the target operator.

[0008] Optionally, copying the calculation parameters corresponding to the target operator of each layer of the target model in the first computing device and the first result parameters to the second computing device includes: Backing up the calculation parameter and the first result parameter in the first calculation device to obtain backup data; sending the backup data to the second computing device; generating a parameter file in a preset format according to the backup data in the second computing device; The parameter file is saved in the second computing device.

[0009] Optionally, the performing, by the deep learning framework in the second computing device, the same computing steps as those of the target operator according to the computing parameters to obtain a second result parameter includes: Reading the parameter file obtained from the first computing device through the second computing device; Acquire the calculation parameters from the parameter file; The calculation parameters are input into the deep learning framework, and the same calculation steps as the target operator are performed to obtain the second result parameters.

[0010] Optionally, comparing intermediate results of the calculation steps of the target operator in the first computing device and the second computing device to determine the error position in the target model includes: Determine all computing steps executed by the target operator in the first computing device and the second computing device; Obtaining first intermediate results of all computing steps corresponding to the target operator in the first computing device; Obtaining second intermediate results of all computing steps corresponding to the target operator in the second computing device; Make a one-to-one correspondence between all the calculation steps in the first calculation device and all the calculation steps in the second calculation device, so as to obtain a correspondence between the calculation steps in the first calculation device and the second calculation device; Comparing the first intermediate result with the second intermediate result according to the corresponding relationship of the calculation steps; When an error between the first intermediate result and the second intermediate result exceeds a preset second error threshold, determining the calculation steps corresponding to the first intermediate result and the second intermediate result; Determining error locations in the target model is the calculating step.

[0011] Optionally, the performing error type judgment on the error in the target model to determine the error type corresponding to the error includes: In the case where an erroneous parameter is loaded in the calculation step of the target model, determining the error type as a loading parameter error; In the case where a calculation error occurs in the calculation step of the target type, the error type is determined to be a calculation error.

[0012] A second aspect of an embodiment of the present application provides an error location device, the device comprising: A parameter copy module, used to copy the calculation parameters corresponding to the target operator of each layer of the target model in the first computing device and the first result parameters to the second computing device; A second result parameter acquisition module, configured to execute the same calculation steps as those of the target operator according to the calculation parameters through a deep learning framework in the second computing device to obtain a second result parameter; An error position determination module is used to compare intermediate results of the calculation steps of the target operator in the first computing device and the second computing device when the error value between the first result parameter and the second result parameter exceeds a preset first error threshold, so as to determine the error position in the target model.

[0013] Optionally, the device further comprises: An error type judgment module, used to judge the error type of the error in the target model and determine the error type corresponding to the error; The error report generating module is used to generate an error report including the error type.

[0014] Optionally, the device further comprises: A main operator determination module, used to determine the main operators included in each layer of the target model according to the structure of the target model; The target operator determination module is used to take the main operator as the target operator.

[0015] Optionally, the data copy module includes: A data backup submodule, used to back up the calculation parameters and the first result parameters in the first computing device to obtain backup data; A data sending submodule, used for sending the backup data to the second computing device; A parameter file generating submodule, used to generate a parameter file in a preset format according to the backup data in the second computing device; The parameter file saving submodule is used to save the parameter file to the second computing device.

[0016] Optionally, the second result parameter acquisition module includes: A parameter file reading submodule, configured to read the parameter file obtained from the first computing device through the second computing device; A calculation parameter acquisition submodule, used to obtain the calculation parameters from the parameter file; The second result parameter acquisition submodule is used to input the calculation parameters into the deep learning framework, execute the same calculation steps as the target operator, and obtain the second result parameters.

[0017] Optionally, the error location determination module includes: A calculation step determination submodule, used to determine all calculation steps executed by the target operator in the first calculation device and the second calculation device; A first intermediate result acquisition submodule, used to acquire first intermediate results of all calculation steps corresponding to the target operator in the first computing device; A second intermediate result acquisition submodule, used to acquire second intermediate results of all calculation steps corresponding to the target operator in the second computing device; A calculation step correspondence submodule, used to make a one-to-one correspondence between all the calculation steps in the first calculation device and all the calculation steps in the second calculation device, so as to obtain a correspondence between the calculation steps in the first calculation device and the second calculation device; A result comparison submodule, used for comparing the first intermediate result with the second intermediate result according to the corresponding relationship of the calculation steps; an error step determination submodule, configured to determine the calculation steps corresponding to the first intermediate result and the second intermediate result when an error between the first intermediate result and the second intermediate result exceeds a preset second error threshold; The error position determination submodule is used to determine the error position in the target model as the calculation step.

[0018] Optionally, the error type judgment module includes: A first error type determination submodule, configured to determine that the error type is a loading parameter error when an erroneous parameter is loaded in the calculation step of the target model; The second error type determination submodule is used to determine that the error type is a calculation error when a calculation error occurs in the calculation step of the target type.

[0019] A third aspect of an embodiment of the present application provides a readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps in the method described in the first aspect of the present application are implemented.

[0020] A fourth aspect of an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the method described in the first aspect of the present application are implemented.

[0021] Using the error location method provided in the present application, the calculation parameters and first result parameters corresponding to the target operator of each layer of the target model in the first computing device are copied to the second computing device; through the deep learning framework in the second computing device, the same calculation steps as the target operator are executed according to the calculation parameters to obtain the second result parameters; when the error value between the first result parameter and the second result parameter exceeds a preset first error threshold, the intermediate results of the calculation steps of the target operator in the first computing device and the second computing device are compared to determine the error position in the target model.

[0022] In this method, a target operator in a target model in a computing device is determined, and then the calculation parameters of the target operator are copied to another computing device as calculation result parameters. The calculation parameters are calculated through a deep learning framework, and then the calculation results in the two devices are compared, so as to quickly determine whether an error occurs in the large language model. When the calculation results differ greatly, the calculation results of each intermediate step of the operator can be compared, so as to quickly locate the position where the error occurs in the large language model, thereby improving the efficiency of error location of the large language model and reducing the cost of debugging and maintenance of the large language model. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0024] Figure 1 is a flow chart of an error location method proposed in an embodiment of the present application; Figure 2 is a schematic diagram of an error location process proposed in an embodiment of the present application; Figure 3 is a schematic diagram of an error locating device proposed in an embodiment of the present application; Figure 4 It is a schematic diagram of an electronic device proposed in an embodiment of the present application. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0026] refer to Figure 1 , Figure 1 XX is a flow chart of the method XX proposed in one embodiment of the present application. Figure 1 As shown, the method comprises the following steps: S11: Copy the calculation parameters and first result parameters corresponding to the target operator of each layer of the target model in the first computing device to the second computing device.

[0027] In this embodiment, the first computing device is a computing device running a target model, and the second computing device is another computing device used to verify the calculation in the first computing device. The target model is a large language model (LLM), which refers to a large-scale deep learning model containing billions or even hundreds of billions of parameters, which is widely used in natural language processing, speech recognition and other fields. This type of model has extremely high computational complexity and requires high performance of the hardware platform. A large language model usually includes multiple layers, each layer includes multiple main operators. In the reasoning process of the large language model, it is usually divided into two main stages, the prefill stage and the generate stage. These two stages are mainly calculated based on the Transformer framework. The operators included in the transformer framework itself include the multi-head attention (MHA) operator, which is a key computing logic in large language models. It captures long-distance dependencies in the input sequence through multiple parallel attention mechanisms. MHA is the key to model performance and is also prone to calculation errors, multi-layer perceptron (MLP) operators, word embedding operators and the final dot product operator. The target operator is the key operator included in each layer of the model, which refers to the key calculation logic that has a greater impact on performance in large language models and is prone to calculation errors, such as MHA and feedforward networks. The calculation parameters include input parameters, weight parameters, kvcache parameters, model shape and other parameters when calculating the operator. The first calculation result parameter is the calculation result obtained after calculating the target operator.

[0028] With the rapid development of artificial intelligence, large language models based on deep learning are increasingly widely used in various fields. Large models usually contain billions or even hundreds of billions of parameters and have extremely high computational complexity and memory requirements.

[0029] In the development process of large model chips, debugging and optimization are one of the key links. Since large models contain a lot of computing logic, such as attention mechanism (MHA), feedforward network, etc., once data calculation errors occur, it is difficult to quickly locate the root cause of the problem. Common problems include: data errors caused by improper upper-level code design; bugs in the underlying hardware implementation; insufficient numerical calculation accuracy, etc.

[0030] Existing large-model chip development and debugging methods usually rely on complex simulation environments, such as using FPGA or RTL simulation platforms to perform comprehensive functional verification and performance analysis of the design. This method can deeply analyze the internal behavior of the chip, but it takes a lot of time and computing resources and it is difficult to quickly locate the problem. It is necessary to build a complex simulation environment, which takes a lot of time and computing resources. There are certain differences between the simulation environment and the actual chip hardware, and it is difficult to completely restore the actual problem. It is difficult to quickly locate the specific location and cause of the problem.

[0031] Another method is to insert a large number of print statements and log records in the chip design, collect detailed information at runtime, and locate the problem by analyzing this information. This method can obtain more fine-grained data, but it needs to process a large amount of log information and is less efficient.

[0032] Some methods focus on monitoring the performance indicators of the chip during operation, such as throughput, latency, power consumption, etc., and infer the root cause of the problem by analyzing these indicators. This method can quickly find performance bottlenecks, but it is difficult to accurately locate the specific location of data errors.

[0033] The existing technical solutions all have certain limitations, and it is difficult to quickly locate the specific location and cause of data errors. It takes a lot of time and resources to complete the debugging process. It is impossible to provide targeted solutions for the characteristics of large model chip development. Therefore, there is an urgent need for a new debugging method that can quickly locate problems, improve debugging efficiency, and optimize for the characteristics of large model chip development. In this embodiment, during the operation of the first computing device, the calculation parameters of the target operator of each layer in the target model during calculation and the first result parameters are sent to the second computing device, and the second computing device then stores the data sent by the first computing device.

[0034] In this embodiment, the specific steps of copying the calculation parameters corresponding to the target operator of each layer of the target model in the first computing device and the first result parameters to the second computing device include: S11 - 1 : Backing up the calculation parameters and the first result parameters in the first calculation device to obtain backup data.

[0035] In this embodiment, when locating errors in the target model in the first computing device, firstly, the corresponding computing parameters and the first result parameters in the target operator of each layer are backed up to obtain backup data.

[0036] S11-2: Send the backup data to the second computing device.

[0037] In this embodiment, after backing up the data, the first computing device sends the backup data to the second computing device.

[0038] S11 - 3: Generate a parameter file in a preset format according to the backup data in the second computing device.

[0039] In this embodiment, the parameter file is a file generated by integrating the calculation parameters and result parameters in the first calculation device.

[0040] In this embodiment, after receiving the backup data, the second computing device generates a parameter file in a corresponding format according to the backup data. The format is preset and is not specifically limited, and is convenient for storage.

[0041] For example, the preset format is a bin format.

[0042] S11-4: Save the parameter file to the second computing device.

[0043] In this embodiment, after generating the parameter file, the second computing device saves the parameter file to the second computing device, and the file can be called at any time when performing calculations to read the parameters therein for performing calculations.

[0044] S12: Using the deep learning framework in the second computing device, execute the same computing steps as the target operator according to the computing parameters to obtain a second result parameter.

[0045] In this embodiment, the deep learning framework is a framework for executing operator calculations, providing an environment required for calculations, and the second result parameter is a result parameter obtained by the second computing device through the input parameters and weight parameters in the target model in the first computing device, executing the same calculation steps as the operator of the target model.

[0046] In this embodiment, the stored parameter file is read by the second computing device, and the input parameters and weight parameters in the parameter file are input into the deep learning framework. The deep learning framework calls the corresponding script to perform the same calculation as the target operator in the first computing device, thereby obtaining the second result parameter.

[0047] By way of example, the deep learning framework is PyTorch.

[0048] In this embodiment, the specific steps of calculating the calculation parameters by the deep learning framework in the second computing device to obtain the second result parameters include: S12-1: Reading the parameter file obtained from the first computing device through the second computing device.

[0049] In this embodiment, when the second computing device executes the same calculation in the first computing device, it first reads the parameter file corresponding to the calculation parameters in the first computing device.

[0050] S12-2: Acquire the calculation parameters from the parameter file.

[0051] In this embodiment, the second computing device obtains corresponding computing parameters from the parameter file.

[0052] S12-3: Input the calculation parameters into the deep learning framework, execute the same calculation steps as the target operator, and obtain the second result parameters.

[0053] In this embodiment, the calculation parameters are input into the deep learning framework, and the same operator calculation in the target model is performed in the deep learning framework to obtain the second result parameters.

[0054] S13: When the error value between the first result parameter and the second result parameter exceeds a preset first error threshold, intermediate results of the calculation steps of the target operator are compared in the first computing device and the second computing device to determine the error position in the target model.

[0055] In this embodiment, after the second computing device performs the same calculation as the target model in the first computing device, a second result parameter is obtained, and the second result parameter is compared with the first result parameter to obtain an error value between the first result parameter and the second result parameter. When the error value between the first result parameter and the second result parameter exceeds a preset first error threshold, the intermediate results of the calculation steps of the target operator in the first computing device and the second computing device are compared to determine the error position in the target model.

[0056] For example, during the error calculation, the root mean square error between the first result parameter and the second result parameter may be calculated, the first error threshold may be set to 10^-3, and the error threshold may be adjusted according to the specific circumstances of different models and different operators.

[0057] In this embodiment, the specific steps of comparing intermediate results of the calculation steps of the target operator in the first computing device and the second computing device to determine the error position in the target model include: S13-1: Determine all computing steps executed by the target operator in the first computing device and the second computing device.

[0058] S13-2: Obtain a first intermediate result of all calculation steps corresponding to the target operator in the first computing device.

[0059] In this embodiment, the first intermediate result is the calculation results of all intermediate steps obtained when the first computing device calculates the target operator.

[0060] In this embodiment, firstly, the first intermediate results of all the calculation steps corresponding to the target operator in the first computing device are obtained.

[0061] For example, the intermediate result may be a calculation result of the attention weight calculation step.

[0062] S13-3: Obtain a second intermediate result of all calculation steps corresponding to the target operator in the second computing device.

[0063] In this embodiment, the second intermediate results of all the calculation steps corresponding to the final target operator of the second computing device are obtained.

[0064] S13-4: Make a one-to-one correspondence between all the calculation steps in the first calculation device and all the calculation steps in the second calculation device to obtain a correspondence between the calculation steps in the first calculation device and the second calculation device.

[0065] In this embodiment, there are multiple calculation steps when calculating the target operator in the first computing device, and there are also multiple calculation steps when the second computing device performs the same calculation on the target operator. When comparing intermediate results, it is necessary to first match all the calculation steps in the first computing device with all the calculation steps in the second computing device one by one, and then obtain the correspondence between each calculation step in the first computing device and the second computing device.

[0066] S13-5: According to the corresponding relationship between the calculation steps, the first intermediate result is compared with the second intermediate result.

[0067] In this embodiment, after determining the correspondence between each calculation step in the first calculation device and each calculation step in the second calculation device, the first intermediate result is compared with the second intermediate result to compare whether there is an error between the calculation results of each calculation step.

[0068] S13-6: When the error between the first intermediate result and the second intermediate result exceeds a preset second error threshold, determining the calculation steps corresponding to the first intermediate result and the second intermediate result.

[0069] In this embodiment, when the error between the first intermediate result and the second intermediate result exceeds the preset second error threshold, it means that the error between the intermediate result obtained after the first computing device executes the calculation step and the intermediate result obtained after the second computing device executes the calculation step is large. At this time, the calculation steps corresponding to the first intermediate result and the second intermediate result are determined.

[0070] S13-7: Determining the error position in the target model is the calculation step.

[0071] In this embodiment, after determining the calculation step corresponding to the intermediate result with a larger error, the location where the error occurs in the target model is determined to be the calculation step.

[0072] For example, the calculation steps of the MHA operator are decomposed into multiple calculation steps, and the intermediate results are verified after each step. For example, after the attention weight calculation step, the system compares the first intermediate result with the second intermediate result. Through comparison, it is found that there are significant differences in the calculation results obtained in the attention weight calculation link, which indicates that there may be problems in the attention weight calculation link.

[0073] In another embodiment of the present application, calculation steps that are given priority for comparison can be pre-set. These calculation steps may have a large amount of calculation and are prone to calculation errors, or they may be calculation steps that have often had errors before. These steps can be compared first. If the error is found directly here, the comparison time is saved, and the error position can be located more quickly. If no error is found in the calculation steps that are given priority for comparison, the calculation steps adjacent to the calculation step can be compared. This can save comparison time.

[0074] In another embodiment of the present application, the method further includes: S21: Determine the error type of the error in the target model and determine the error type corresponding to the error.

[0075] In this embodiment, after determining the calculation step where the error in the target model occurs, it is also necessary to determine the error type of the error that occurs in the target model.

[0076] In this embodiment, the specific steps of judging the error type of the error in the target model and determining the error type corresponding to the error include: S21-1: When an erroneous parameter is loaded in the calculation step of the target model, determining that the error type is a loading parameter error.

[0077] In this embodiment, when an erroneous parameter is loaded in the calculation step of the target model, for example, an error occurs in loading a weight parameter, which will cause a large deviation in the calculation result. Therefore, the error type is determined to be a loading parameter error.

[0078] S21-2: When a calculation error occurs in the calculation step of the target type, determine that the error type is a calculation error.

[0079] In this embodiment, if the loaded parameters in the calculation step of the target model are correct, but the error of the calculation result is large, it means that a calculation error has occurred in the calculation step, and then the error type is determined to be a calculation error.

[0080] S22: Generate an error report including the error type.

[0081] In this embodiment, after the error type occurring in the calculation step is determined, an error report including the error type is generated.

[0082] In this embodiment, the error report must include the error location of the error occurring in the target model, the error type of the error, and also includes the expected output result, the actual output result, and the error between the two output results.

[0083] In this embodiment, the error report will also generate corresponding suggestions based on the error location and error type of the target model, suggesting corresponding processing for the step where the error occurred.

[0084] For example, the system generates a report indicating that there is a problem in the calculation of attention weights, lists the expected output results, the actual output results, and the error between the two, and recommends checking the loading process of weight parameters and the calculation logic of attention weights.

[0085] In this embodiment, multiple target operators of each layer are verified through the above steps, and then the location where the error occurs in the target model is determined.

[0086] In another embodiment of the present application, before copying the calculation parameters and the first result parameters corresponding to the target operator of each layer of the target model in the first computing device to the second computing device, the method further includes: S31: According to the structure of the target model, determine the main operators included in each layer of the target model.

[0087] In this embodiment, each model includes multiple layers, and operator operations need to be performed in each layer. Each layer in the target model also includes multiple operator operations. According to the structure of the target model, the main operators included in each layer of the target model can be determined.

[0088] For example, large language model calculations usually include multi-head attention operators, multi-layer perceptron operators, word embedding operators, and dot product operators.

[0089] S32: Taking the main operator as the target operator.

[0090] In this embodiment, the main operator of each layer in the target model is used as the target operator.

[0091] In another embodiment of the present application, reference Figure 2 , Figure 2 is a schematic diagram of an error location process proposed in an embodiment of the present application, such as Figure 2 As shown, when the error location starts, the procedure loves you to split the operators of the large language model into multi-head attention operator (MHA), multi-layer perceptron operator (MLP), word embedding operator (Embedding) and dot product operator (dot) according to the structure of the large language model. For each operator, the input data, output data, weight, kvcache parameters and other data of the operator are obtained in the first computing device, and the obtained data is copied to the second computing device. The operation of the large language model is continued to be performed in the first computing device, and the obtained data is input into the deep learning framework in the second computing device for operation. The calculation results of the two are compared to determine whether the error between the calculation results exceeds the preset first error threshold. If it exceeds the threshold, the error position of the large language model is quickly located. If it does not exceed the threshold, it is determined that the calculation result in the first computing device is correct.

[0092] In this embodiment, after determining the error position of the target model in the first computing device, the program and parameters of the target model in the first computing device are adjusted according to the suggestions given in the system report, and the test is continued after the adjustment is completed until the error value between the first result parameter and the second result parameter in the first computing device does not exceed the preset first error threshold, and the debugging is completed.

[0093] In the above embodiments of the present application, the calculation process of the target operator is accurately located during the reasoning process of the large language model, thereby improving the efficiency of fault diagnosis. The front-end developers do not need to understand the implementation method and syntax of the underlying operators to independently locate the problems in the model calculation, saving the time spent by the developers, reducing the difficulty of locating the model calculation problems, and making the application of the model easier and more extensive. By using a deep learning framework in another computing device to recalculate the key operators and comparing them with the calculation results in the original computing device, the accuracy of the reasoning results is ensured. After the operator problem is discovered, the root cause of the problem is quickly located by splitting the calculation steps. Whether it is an upper-level scheduling problem or a bottom-level implementation problem, it can be optimized in a targeted manner to avoid waste of resources. By accurately locating the problem, the time cost of debugging and maintaining the model is reduced, and the development efficiency of the large language model is improved.

[0094] In the above embodiments of the present application, a mechanism for quickly locating and solving problems is established. Once it is found that the error of the calculation results between two computing devices exceeds the error threshold, the problem can be quickly located and fed back by generating a report, thereby accelerating the diagnosis and resolution of the problem and improving the optimization efficiency of the model. This process minimizes human intervention and reduces the waste of resources caused by incorrect diagnosis. Through continuous cycles, the process can continuously optimize itself to adapt to the ever-changing models and application requirements, and can be flexibly applied to AI inference chips with different architectures.

[0095] Based on the same inventive concept, an embodiment of the present application provides an error locating device. Figure 3 , Figure 3 FIG. 3 is a schematic diagram of an error location device 300 proposed in an embodiment of the present application. Figure 3 As shown, the device comprises: A parameter copy module 301, used to copy the calculation parameters corresponding to the target operator of each layer of the target model in the first computing device and the first result parameters to the second computing device; A second result parameter acquisition module 302 is used to perform the same calculation steps as the target operator according to the calculation parameters through the deep learning framework in the second computing device to obtain a second result parameter; The error position determination module 303 is used to compare the intermediate results of the calculation steps of the target operator in the first computing device and the second computing device when the error value between the first result parameter and the second result parameter exceeds a preset first error threshold, so as to determine the error position in the target model.

[0096] Optionally, the device further comprises: An error type judgment module, used to judge the error type of the error in the target model and determine the error type corresponding to the error; The error report generating module is used to generate an error report including the error type.

[0097] Optionally, the device further comprises: A main operator determination module, used to determine the main operators included in each layer of the target model according to the structure of the target model; The target operator determination module is used to take the main operator as the target operator.

[0098] Optionally, the data copy module includes: A data backup submodule, used to back up the calculation parameters and the first result parameters in the first computing device to obtain backup data; A data sending submodule, used for sending the backup data to the second computing device; A parameter file generating submodule, used to generate a parameter file in a preset format according to the backup data in the second computing device; The parameter file saving submodule is used to save the parameter file to the second computing device.

[0099] Optionally, the second result parameter acquisition module includes: A parameter file reading submodule, configured to read the parameter file obtained from the first computing device through the second computing device; A calculation parameter acquisition submodule, used to obtain the calculation parameters from the parameter file; The second result parameter acquisition submodule is used to input the calculation parameters into the deep learning framework, execute the same calculation steps as the target operator, and obtain the second result parameters.

[0100] Optionally, the error location determination module includes: A calculation step determination submodule, used to determine all calculation steps executed by the target operator in the first calculation device and the second calculation device; A first intermediate result acquisition submodule, used to acquire first intermediate results of all calculation steps corresponding to the target operator in the first computing device; A second intermediate result acquisition submodule, used to acquire second intermediate results of all calculation steps corresponding to the target operator in the second computing device; A calculation step correspondence submodule, used to make a one-to-one correspondence between all the calculation steps in the first calculation device and all the calculation steps in the second calculation device, so as to obtain a correspondence between the calculation steps in the first calculation device and the second calculation device; A result comparison submodule, used for comparing the first intermediate result with the second intermediate result according to the corresponding relationship of the calculation steps; an error step determination submodule, configured to determine the calculation steps corresponding to the first intermediate result and the second intermediate result when an error between the first intermediate result and the second intermediate result exceeds a preset second error threshold; The error position determination submodule is used to determine the error position in the target model as the calculation step.

[0101] Optionally, the error type judgment module includes: A first error type determination submodule, configured to determine that the error type is a loading parameter error when an erroneous parameter is loaded in the calculation step of the target model; The second error type determination submodule is used to determine that the error type is a calculation error when a calculation error occurs in the calculation step of the target type.

[0102] Based on the same inventive concept, another embodiment of the present application provides a readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the error location method described in any of the above embodiments of the present application.

[0103] Based on the same inventive concept, another embodiment of the present application provides an electronic device, Figure 4 It is a schematic diagram of an electronic device 400 proposed in an embodiment of the present application, including a memory 402, a processor 401 and a computer program stored in the memory and executable on the processor, and when the processor executes, the steps in the error location method described in any of the above embodiments of the present application are implemented.

[0104] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0105] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0106] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the embodiments of the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0107] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0108] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0109] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0110] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0111] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0112] The error location method, device, equipment and storage medium provided by the present application are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. An error location method, characterized in that: The method comprises: Copying the calculation parameters and the first result parameters corresponding to the target operator of each layer of the target model in the first computing device to the second computing device; By using the deep learning framework in the second computing device, the same computing steps as those of the target operator are executed according to the computing parameters to obtain a second result parameter; When the error value between the first result parameter and the second result parameter exceeds a preset first error threshold, intermediate results of the calculation steps of the target operator are compared in the first computing device and the second computing device to determine the error position in the target model.

2. The error location method according to claim 1, characterized in that: The method further comprises: Performing error type judgment on the error in the target model to determine the error type corresponding to the error; An error report is generated including the error type.

3. The error location method according to claim 1, characterized in that: Before copying the calculation parameters and the first result parameters corresponding to the target operator of each layer of the target model in the first computing device to the second computing device, the method further includes: According to the structure of the target model, determining the main operators included in each layer of the target model; The main operator is used as the target operator.

4. The error location method according to claim 1, characterized in that: The step of copying the calculation parameters and the first result parameters corresponding to the target operator of each layer of the target model in the first computing device to the second computing device includes: Backing up the calculation parameter and the first result parameter in the first calculation device to obtain backup data; sending the backup data to the second computing device; generating a parameter file in a preset format according to the backup data in the second computing device; The parameter file is saved in the second computing device.

5. The error location method according to claim 1, characterized in that: The step of performing the same calculation steps as the target operator according to the calculation parameters through the deep learning framework in the second computing device to obtain a second result parameter includes: Reading the parameter file obtained from the first computing device through the second computing device; Acquire the calculation parameters from the parameter file; The calculation parameters are input into the deep learning framework, and the same calculation steps as the target operator are performed to obtain the second result parameters.

6. The error location method according to claim 1, characterized in that: The comparing intermediate results of the calculation steps of the target operator in the first computing device and the second computing device to determine the error position in the target model includes: Determine all computing steps executed by the target operator in the first computing device and the second computing device; Obtaining first intermediate results of all computing steps corresponding to the target operator in the first computing device; Obtaining second intermediate results of all computing steps corresponding to the target operator in the second computing device; Make a one-to-one correspondence between all the calculation steps in the first calculation device and all the calculation steps in the second calculation device, so as to obtain a correspondence between the calculation steps in the first calculation device and the second calculation device; Comparing the first intermediate result with the second intermediate result according to the corresponding relationship of the calculation steps; When an error between the first intermediate result and the second intermediate result exceeds a preset second error threshold, determining the calculation steps corresponding to the first intermediate result and the second intermediate result; Determining error locations in the target model is the calculating step.

7. The error location method according to claim 2, characterized in that: The step of judging the error type of the error in the target model and determining the error type corresponding to the error includes: In the case where an erroneous parameter is loaded in the calculation step of the target model, determining the error type as a loading parameter error; In the case where a calculation error occurs in the calculation step of the target type, the error type is determined to be a calculation error.

8. An error locating device, characterized in that: The device comprises: A parameter copy module, used to copy the calculation parameters corresponding to the target operator of each layer of the target model in the first computing device and the first result parameters to the second computing device; A second result parameter acquisition module, configured to execute the same calculation steps as those of the target operator according to the calculation parameters through a deep learning framework in the second computing device to obtain a second result parameter; An error position determination module is used to compare intermediate results of the calculation steps of the target operator in the first computing device and the second computing device when the error value between the first result parameter and the second result parameter exceeds a preset first error threshold, so as to determine the error position in the target model.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps in the method according to any one of claims 1 to 7 are implemented.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

Images