Abnormal operator positioning method and device, storage medium and program product

Abstract

The invention discloses an abnormal operator positioning method and device, a storage medium and a program product, and relates to the technical field of computers, and the method comprises the steps: rapidly locking the candidate range of an abnormal operator through comparing the operation result difference of a target model in different computing devices, and avoiding the resource waste of total operator troubleshooting; dynamically generating a to-be-verified operator set according to the operators called by the target model, and compressing the troubleshooting range from the whole operator set to a set corresponding to the operators called by the target model; the technical problem of low efficiency in a manual layer-by-layer verification method is solved by returning to verification computing equipment one by one and combining automatic result comparison and accurate positioning of abnormal operators, and the technical effects of improving the troubleshooting efficiency and avoiding the risk of manual misoperation are achieved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, storage medium and program product for locating an abnormal operator. Background Art

[0002] With the rapid development of large models in the field of artificial intelligence, the number of large models continues to increase, and the types of hardware devices running these models are also becoming increasingly rich. Different hardware devices need to adapt to different kernels (parallel computing functions), and different large models correspond to different operators. Even for the same operator, their inputs may be different. During the training or reasoning process of large models, model anomalies such as abnormal training loss or incorrect reasoning results may occur, and these anomalies are usually caused by abnormal operator calculations. Therefore, screening out incorrect operators and corresponding inputs and outputs is crucial to locating and solving model anomalies. At present, a more common method in related technologies is to verify the large model layer by layer from the front end of PyTorch (an open source deep learning framework), and find the problem by comparing the results of each operator. This method has the problem of low efficiency. Summary of the invention

[0003] The present application provides a method, device, storage medium and program product for locating abnormal operators, so as to at least solve the technical problem of low efficiency existing in the manual layer-by-layer verification method in the related technology, thereby achieving the technical effect of improving the troubleshooting efficiency and avoiding the risk of manual operation errors.

[0004] The present application provides a method for locating an abnormal operator, comprising: obtaining the running results of a target model on at least two different computing devices; when determining that an abnormal operator exists according to the difference in the running results, generating a list of operators to be verified according to the operators called during the running of the target model; the abnormal operator is at least one operator called during the running of the target model; the operators in the list of operators to be verified are rolled back to a verification computing device one by one, so that the verification computing device runs the target model according to the rolled back operators to obtain a verification result; and locating the abnormal operator according to the verification result and the running result.

[0005] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-mentioned abnormal operator locating methods when executing the computer program.

[0006] The present application also provides a computer-readable storage medium, in which a computer program is stored, wherein when the computer program is executed by a processor, the steps of any of the above-mentioned abnormal operator locating methods are implemented.

[0007] The present application also provides a computer program product, including a computer program, which implements the steps of any of the above-mentioned abnormal operator locating methods when the computer program is executed by a processor.

[0008] Through this application, by comparing the differences in the running results of the target model on different computing devices, the candidate range of abnormal operators can be quickly locked to avoid the waste of resources for the full operator screening; according to the operators called by the target model, a set of operators to be verified is dynamically generated, and the screening range is compressed from the entire operator set to the set corresponding to the operators called by the target model; by returning to the verification computing device one by one, combined with the automatic result comparison, the abnormal operator can be accurately located. Therefore, the technical problem of low efficiency in the manual layer-by-layer verification method can be solved, and the technical effect of improving the screening efficiency and avoiding the risk of manual operation errors can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0010] Figure 1 A flowchart of a method for locating an abnormal operator provided in an embodiment of the present application;

[0011] Figure 2 A schematic diagram of a module of a method for locating an abnormal operator provided in an embodiment of the present application;

[0012] Figure 3 A schematic diagram of a scheduling method for locating an abnormal operator provided in an embodiment of the present application;

[0013] Figure 4 A specific flow chart of a method for locating an abnormal operator provided in an embodiment of the present application;

[0014] Figure 5 A schematic diagram of an electronic device provided in an embodiment of the present application;

[0015] Figure 6 A schematic diagram of a computer-readable storage medium provided in an embodiment of the present application. DETAILED DESCRIPTION

[0016] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0017] It should be noted that, in the description of this application, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. The terms "first", "second", etc. in this application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0018] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0019] The embodiment of the present application provides a method for locating an abnormal operator, and the method is described in detail in combination with the execution flow of the method for locating an abnormal operator. In computer science, kernel refers to a parallel computing function running on a GPU (Graphics Processing Unit). PyTorch is an open source deep learning framework designed for deep learning and artificial intelligence research. In deep learning, an operator is like an operator in mathematics, which is a basic building block for performing specific operations. An operator can be imagined as a "black box" that receives some input data, processes the data according to predefined rules, and finally outputs the processed results; the operator is one of the core components for building a neural network. Different operators can be combined to form a complex calculation graph, thereby realizing the functions of various deep learning models. For example, a simple neural network layer may be composed of a matrix multiplication operator and an activation function operator. Common operator types include: matrix multiplication, convolution, addition, subtraction, multiplication and division, activation function, pooling, normalization, etc. By combining these operators, deep learning models can perform complex tasks such as image recognition, natural language processing, and speech recognition.

[0020] like Figure 1 As shown, the method for locating the abnormal operator includes:

[0021] S11: Obtain the running results of the target model on at least two different computing devices.

[0022] In this step, potential computational anomalies are detected by comparing the running results of the target model on different computing devices. In practical applications, different hardware platforms (such as GPU, CPU (Central Processing Unit) or other computing devices) may produce different computational results due to differences in computing architecture, accuracy, optimization, etc. By running the target model on at least two different computing devices (such as GPU and TPU (Tensor Processing Unit) (TPU is the remaining computing devices except GPU and CPU)) and obtaining the computational results under the same input, a strong basis can be provided for the subsequent location of operator anomalies. If there are significant differences in the results of different computing devices, then this difference may indicate that some operators have computational anomalies.

[0023] S12: When it is determined that there is an abnormal operator according to the difference of the running results, a list of operators to be verified is generated according to the operators called during the running of the target model; the abnormal operator is at least one operator called during the running of the target model.

[0024] When the target model is running on different computing devices, if the difference in the output results is detected to exceed the predetermined threshold (i.e., the difference is large), it means that an abnormality may have occurred during the operation of the target model. At this time, it is necessary to locate the operator that may cause this abnormality.

[0025] To achieve this, it is necessary to list all operators called by the target model during operation. These operators are the basic components of the target model, and each operator has a direct or indirect impact on the output of the target model. A list of operators to be verified is generated based on all the called operators. These operators will be further verified and checked to determine which operator caused the model anomaly.

[0026] S13: Return the operators in the list of operators to be verified to the verification computing device one by one, so that the verification computing device runs the target model according to the returned operators to obtain the verification result.

[0027] After the list of operators to be verified is determined, these operators in the list of operators to be verified are rolled back to the verification computing device one by one. Rolling back to the verification computing device means switching the operator to be verified from the current computing device to another device and running the target model on the verification computing device. The verification computing device will perform the calculation of the target model based on the rolled back operator and generate the corresponding verification result. The purpose of this step is to confirm whether there is a problem with the calculation of the operator by verifying the execution of the same operator by the computing device under different hardware conditions.

[0028] S14: Locate the abnormal operator based on the verification results and operation results.

[0029] By comparing the verification results with the original running results, determine which operators have anomalies. During the verification phase, the verification results obtained by the verification computing device will be compared with the results of the initial run of the target model. In this way, the operator that causes the target model to run abnormally can be located. This automatic positioning process can quickly and efficiently identify abnormal operators, thereby reducing the time and cost of manual debugging and improving the efficiency of abnormality detection.

[0030] In an exemplary embodiment, the operation results of a target model on at least two different computing devices are obtained, including: running the target model on a first computing device to generate a baseline result; running the target model on a second computing device to generate a candidate result; the operation results include the baseline result and the candidate result; determining the presence of an abnormal operator based on the difference in the operation results, including: judging whether the difference between the candidate result and the baseline result meets a preset difference condition; if so, determining that an abnormal operator exists when the second computing device runs the target model.

[0031] In this embodiment, the location of abnormal operators is achieved by comparing the differences in the running results of the target model on heterogeneous computing devices. Specifically, the target model is run on the first computing device (such as a GPU with verified correct results) to generate a benchmark result as a reliable reference for subsequent comparisons; the same target model is run on the second computing device (such as a TPU to be verified) to generate candidate results, and the difference between the two is quantified based on indicators such as tensor element-by-element error, statistical distribution difference, or model function output consistency; the difference can be an error in the output value, a difference in calculation accuracy, or a more complex structural difference. If the difference meets the preset difference condition, it is determined that the second computing device has an anomaly caused by an error in the implementation of a specific operator.

[0032] It can be seen that this embodiment can effectively locate the operator that has an abnormality when executed on the second computing device by detecting and quantifying such differences, thereby providing a basis for further abnormality analysis and troubleshooting.

[0033] In an exemplary embodiment, the first computing device is a reference computing device whose operation results have been verified, and the second computing device is a computing device to be verified.

[0034] Specifically, the first computing device (such as a verified GPU) is used as a reference device, whose operator implementation has been fully verified and the results are reliable, and is used to generate the benchmark results of the target model; the second computing device (such as the TPU to be verified) is used as a target device, whose operator implementation may have potential defects, and runs the same target model to generate candidate results. The difference between the two comes from the specific implementation of the operator by the hardware (such as numerical accuracy, parallel optimization or memory management). By comparing the difference between the benchmark results and the candidate results, it can be determined whether there are abnormal operators on the second computing device. Specifically, if the difference between the candidate result and the benchmark result exceeds a preset threshold (such as the element-by-element error exceeds the preset threshold or the functional output is inconsistent), it indicates that there is a computational anomaly in the operator implementation of the second computing device, and it is necessary to further isolate the specific problem operator through a dynamic fallback mechanism, so as to converge the complex model anomaly to the local defect of hardware adaptation.

[0035] In an exemplary embodiment, determining whether the difference between the candidate result and the benchmark result satisfies a preset difference condition includes: determining whether the difference between the candidate result and the benchmark result is greater than a preset threshold; if not, determining that the preset difference condition is satisfied.

[0036] In this embodiment, the process of determining whether the difference between the candidate result and the benchmark result meets the preset difference condition is mainly to determine whether there is an abnormality by comparing the difference between the running results on the two computing devices. Specifically, the difference between the candidate result and the benchmark result is calculated, and this difference can be expressed as a numerical error, a result deviation, or other forms of calculation difference, and then the difference is compared with a preset threshold.

[0037] If the difference between the candidate result and the benchmark result is less than the preset threshold, then it can be determined that the calculation results of the two are very close, that is, the difference on the computing device is acceptable and meets the expected calculation accuracy range, so there is no need to further locate the operator anomaly. At this time, the result of the judgment is that the preset difference condition is not met, that is, the difference is small, and it is considered that there is no anomaly.

[0038] On the contrary, if the difference between the candidate result and the benchmark result is greater than the preset threshold, it indicates that there is a significant deviation between the two, indicating that at least one operator on the second computing device is abnormal, resulting in incorrect calculation results. Therefore, it is determined that the preset difference condition is met, indicating that the abnormal operator needs to be further checked.

[0039] In this way, we can effectively screen out which operators may cause abnormal calculation results, and conduct further verification and repair in a targeted manner.

[0040] In an exemplary embodiment, locating an abnormal operator according to the verification result and the operation result includes: locating the abnormal operator according to the difference between the verification result and the benchmark result. For example, in a specific embodiment, locating the abnormal operator according to the difference between the verification result and the benchmark result includes: determining whether the verification result is the same as the benchmark result; if the verification result is the same as the benchmark result, determining that the currently rolled back operator is an abnormal operator; if the verification result is different from the benchmark result, determining that the currently rolled back operator is a normal operator.

[0041] In an exemplary embodiment, a specific implementation method for locating abnormal operators is to roll back each operator in the list to be verified to a verified third computing device (such as a CPU) for execution (other operators remain on the TPU); if the verification result is consistent with the benchmark result after a certain operator is rolled back, it indicates that there is an abnormality in the implementation of the operator on the original device (TPU) (because the result returns to normal after it is rolled back to the CPU), thereby directly locking the operator as an abnormal operator; conversely, if the verification result is still abnormal, the currently rolled back operator is excluded as an abnormal operator, and the next operator is tested.

[0042] This embodiment uses a single variable isolation strategy to converge complex model problems to the hardware adaptation defects of specific operators, thereby achieving efficient and accurate anomaly location.

[0043] In an exemplary embodiment, after generating a list of operators to be verified according to the operators called during the operation of the target model, it also includes: generating a configuration file according to the list of operators to be verified; returning the operators in the list of operators to be verified to the verification computing device one by one, including: parsing the configuration file, and returning the operators in the configuration file to the verification computing device one by one.

[0044] In this embodiment, after a list of operators to be verified is generated according to operators called during the operation of the target model, abnormal operators are further located by generating a configuration file and rolling back operators one by one to a verification computing device.

[0045] The purpose of generating a configuration file (such as a YAML configuration file) is to save the list of operators to be verified in a structured form and ensure that subsequent operations have clear steps. The configuration file can include detailed information about the operators to be verified, such as the string name of each operator, related parameters, information about the verification computing device, and the detailed process of how to roll back these operators to the verification computing device one by one. The configuration file helps to associate the operators to be verified with specific rollback steps or verification computing devices to ensure operability and traceability during the verification process.

[0046] Next, the operators in the list of operators to be verified will be rolled back to the verification computing device one by one. This process requires parsing the configuration file and performing specific operations according to the definitions therein. The operator information in the configuration file is parsed, and the operators are rolled back from the current computing device to the specified verification computing device one by one according to the order indicated in the configuration file and the requirements of the verification computing device. The rollback process actually transfers an operator in the target model from the current running platform (such as TPU) to a known stable platform (such as CPU). By executing the operator on the verification computing device and comparing the difference between the verification result and the benchmark result, it can be determined whether there is a problem with the implementation of the operator on different computing devices.

[0047] The format of the configuration file can be:

[0048] yaml;

[0049] name:UNREGISTER_OPERS;

[0050] value:add,mm,conv,…;

[0051] The name of the yaml configuration file item is UNREGISTER_OPERS, and the value of the item is a list of operators to be verified, such as add, mm, and conv. When registering an operator, dynamically read the UNREGISTER_OPERS information in the yaml configuration file to determine which operators need to fall back to the third computing device for execution. Based on this configuration, use conditional judgment and the RegisterPlugin function to control the registration and fallback of operators.

[0052] The core principle of this process is to manage complex operator rollback steps in a structured manner through configuration files, and to perform rollback verification one by one according to the configuration. In this way, potential problems of each operator can be effectively isolated, avoiding omissions in large-scale model verification, and ensuring the consistency and stability of each operator running on different hardware platforms.

[0053] In an exemplary embodiment, a configuration file is parsed and the operators in the configuration file are rolled back one by one to a verification computing device, including: parsing the configuration file to obtain the string names of the operators in the configuration file; in the process of registering operators, according to the string names of the operators in the configuration file, skipping the registration of the operators in the configuration file, so that the operators in the configuration file are rolled back one by one to the verification computing device when the target model is run.

[0054] In this embodiment, the process of parsing the configuration file and rolling back the operators in the configuration file to the verification computing device one by one is mainly described. Specifically, the purpose of parsing the configuration file can be to extract the string names of the operators contained therein, which are unique identifiers that identify the operators to be verified. The configuration file lists the names of these operators and their related information in a structured form. By parsing these string names, it is possible to know which operators need to be rolled back to the verification computing device. In the process of registering the operator, according to the operator string name in the configuration file, the scheduling layer will skip the registration of the operator in the configuration file according to the predefined rules. Generally, operator registration is an important step in the operation of the model. It registers each operator in the model to the execution environment to ensure that they can be executed as expected. However, in this step, since the goal is to roll back the operators in the configuration file to the verification computing device one by one, rather than executing on the current computing device, it is necessary to skip the registration of these operators. By skipping the registration of the operators in the configuration file, the operators in the configuration file will not be executed on the current device, but will be rolled back to the verification computing device for execution. In this process, the configuration file plays a guiding role, ensuring that each operator is rolled back to the verification computing device in sequence and one by one.

[0055] This mechanism ensures that the verification process of the operator can be carried out one by one in a strictly controlled environment. It not only avoids redundant calculations of the operator on the current device, but also accurately detects potential problems of the operator, ultimately achieving efficient anomaly location.

[0056] In an exemplary embodiment, it also includes: pre-defining an operator registry, the operator registry is used to store the mapping relationship between the string name of the operator and the corresponding execution logic; in the process of registering the operator, according to the string name of the operator in the configuration file, skipping the registration of the operator in the configuration file, including: when filling the operator registry with a preset registration function, skipping the filling of the operator corresponding to the target string name, and filling the operator registry according to other operators other than the operator in the configuration file; the target string name is the string name of the operator in the configuration file; when running the target model, the operators in the configuration file are rolled back to the verification computing device one by one, including: when running the target model, calling the filled operator registry; the filled operator registry does not include the operator in the configuration file; according to the filled operator registry, the operators in the configuration file are rolled back to the verification computing device one by one.

[0057] In this embodiment, the purpose is to flexibly control the registration and rollback operations of operators through the cooperation of the operator registry and the configuration file to realize the verification of the target model. Specifically, an operator registry is first predefined, and the operator registry is used to store the string names of the operators and their corresponding execution logic. The function of the operator registry is to quickly find and execute the operations of the operators through mapping relationships. In the process of registering the operator, the configuration file is first parsed to extract the string names of the operators listed therein, and according to these names, it is determined which operators need to be skipped for registration. Specifically, when the preset registration function is called to fill the operator registry, according to the operator string names in the configuration file, the operators in these configuration files are avoided from being registered in the operator registry, that is, their registration operations are skipped. In this way, only the operators not listed in the configuration file will be normally registered in the operator registry. Among them, the target string name refers to the name of the operator listed in the configuration file. During the registration process, these operators will be skipped and will not be filled in the registry.

[0058] When the target model is run, the populated operator registry is called. Since the operator registry does not contain the operators skipped in the configuration file, these skipped operators will not directly participate in the normal calculation process (that is, they will not be calculated on the second computing device), but will be rolled back to the verification computing device one by one. The purpose of rollback is to migrate the operators specified in the configuration file to the verification computing device for further verification. On the verification computing device, these operators can be independently executed, tested, and verified to ensure the correctness of the calculation in a specific environment.

[0059] Among them, the preset registration function can realize the filling of the operator registry by defining a RegisterPlugin function. In this function, by traversing the operators and their corresponding execution logic, the map data structure is used to fill the registry with the operator's string name as the key, and the corresponding execution logic function pointer as the value. In this way, the RegisterPlugin function dynamically registers all operators and their execution logic into the system, so that the corresponding logic can be efficiently found and executed according to the operator name when the subsequent target model is running. This mechanism simplifies the operator registration process and can be flexibly expanded.

[0060] In an exemplary embodiment, the operator registry is configured as a key-value pair structure, wherein the key is the string name of the operator, and the value is the function corresponding to the execution logic of the operator.

[0061] In this embodiment, the operator registry is designed as a key-value pair structure (such as using std::map to store the operator name and the corresponding execution function mapping, and filling it through the RegisterPlugin function), the key is the string name of the operator, and the value is the function pointer of the execution logic corresponding to the operator. Specifically, the operator string name is used as the key of the registry. In this way, each operator can be identified by the string name, and the execution logic related to the operator can be quickly accessed. The value is a function pointer pointing to the operator execution logic, ensuring that the corresponding function can be called to execute the operator operation when needed.

[0062] When an operator needs to be registered, the operator's string name and corresponding function pointer are added to the operator registry through a registration process. The advantage of this structure is that it can achieve efficient search and call. When an operator needs to be executed when the target model is running, the corresponding execution logic (function pointer) is quickly obtained by looking up the string name (key) in the registry to implement the execution of the operator. This design method avoids defining the same logic code multiple times in the program, and improves the maintainability and scalability of the code.

[0063] For example, if there are multiple operators, such as addition, multiplication, activation functions, etc., each operator has its own corresponding function. The operator registry uses the string names of these operators (such as "Add", "Multiply", "ReLU", etc.) as keys and associates them with the function pointers corresponding to these operators (such as add_func, multiply_func, relu_func). When running the target model, you only need to query the operator registry to find and call the corresponding operator execution logic.

[0064] This operator registry organized in a key-value pair structure not only achieves efficient management and search of operators, but also allows operators to be flexibly registered or uninstalled as needed, enhancing the flexibility and scalability of operator management. In addition, combined with the configuration switch and the mechanism of skipping specific operators, it is possible to accurately control which operators should participate in the calculation and which operators should be excluded at runtime, further optimizing the operator management process.

[0065] It can be seen that this embodiment ensures that the operators in the configuration file can be fully verified without interfering with the normal operation of the target model. The registration and rollback of operators can be flexibly controlled during the verification process, thus avoiding unnecessary interference and improving verification efficiency.

[0066] In an exemplary embodiment, after predefining the operator registry, it also includes: defining a configuration switch corresponding to the operator, the state of the configuration switch represents whether the corresponding operator is filled in the operator registry, and the state of the configuration switch includes an off state and an on state; when filling the operator registry using a preset registration function, skipping the filling of the operator corresponding to the target string name, including: setting the configuration switch corresponding to the target string name to a closed state; when filling the operator registry using a preset registration function, skipping the filling of the operator corresponding to the closed state of the configuration switch, and filling the operator registry according to the operator whose configuration switch state is an open state.

[0067] In this embodiment, in order to further enhance the control and flexibility of the operator registration process, the concept of a configuration switch is introduced, and each operator corresponds to a configuration switch. The state of this configuration switch is used to indicate whether the operator will be filled into the operator registration table. There are two states of the configuration switch, an open state and a closed state. The open state means that the operator will be registered in the operator registration table, while the closed state means that the operator will be skipped and will not be registered in the operator registration table. In this way, it is possible to flexibly control which operators need to participate in the operation of the target model and which operators need to be excluded in order to perform specific verification operations.

[0068] Specifically, when the operator registry is filled using the preset registration function, the state of the configuration switch of each operator is first checked. If the state of the configuration switch is on, the operator will be registered in the operator registry according to the normal process, mapping its string name and the corresponding execution logic. If the state of the configuration switch is off, the operator will be skipped and the preset registration function will ignore its registration. Specifically, when an operator corresponding to the target string name listed in the configuration file is encountered, the state of the configuration switch corresponding to the operator is set to off, so that the operator will not be filled in the operator registry, avoiding its participation in the execution process of the target model on the second computing device.

[0069] It can be seen that in this application, by configuring the switch, it is possible to accurately control which operators should be registered and which operators should be excluded, thereby improving flexibility and configurability. When the target model is actually run, only those operators whose configuration switch status is on will be used and executed, while operators whose configuration switch is off will be skipped to ensure that the model can run as expected. In addition, through this configuration switch control, the operator registration process can be combined with the rollback operation of the verification computing device. Without affecting the overall computing process, the operators that need to be verified will be rolled back to the verification computing device one by one, further improving the verification efficiency.

[0070] In an exemplary embodiment, it also includes: recording the input and output parameter information of the abnormal operator and the string name of the abnormal operator; the input and output parameter information includes the input and output tensor shape, step size and data type; generating a corresponding test script for the operator according to the input and output parameter information, string name and preset script; the test script is used to reproduce the calculation process of the abnormal operator.

[0071] In this embodiment, by recording the input and output parameter information of the abnormal operator and the string name of the operator, the runtime behavior of the operator can be obtained. The input and output parameter information includes the shape, step size, and data type of the tensor. These parameters are important attributes that describe the input and output data of the operator and are essential for locating and analyzing abnormal behavior. By collecting this information, all key factors related to the operator can be taken into account when generating a test script. Next, the operator is generated based on the preset script, and the corresponding test script is automatically generated in combination with the collected input and output parameter information and string name. The test script will simulate the execution of the abnormal operator under various input conditions, verify whether it works as expected, and can capture potential errors or inconsistencies.

[0072] It can be seen that this embodiment not only improves the efficiency of operator verification through this automated script generation method, but also ensures the comprehensiveness and accuracy of the test, thereby effectively detecting and repairing possible defects in the operator.

[0073] like Figure 2As shown in the figure, the automatic model troubleshooting program corresponding to the location of abnormal operators can be divided into the following virtual modules, namely, model running module, operator list processing module, result processing module, and operator test script generation module. Among them, the model running module executes the reasoning or training task of the target model on heterogeneous computing devices (such as GPU, TPU) and generates the running results. For example, the target model is run on a verified device (such as GPU) and the output result is saved as the correct benchmark result; the same target model is run on a device to be verified (such as TPU), the candidate results are recorded, and the operator call tracking function is enabled to generate a list of operators to be verified that are actually called. The operator list processing module dynamically manages the list of operators to be verified called during the operation of the target model, generates and parses the configuration file to control the operator fallback logic; for example, the list to be verified corresponding to the operator name is obtained from the model running module, and the yaml configuration file is automatically generated, marking the operator to be rolled back to the CPU. According to the verification progress, the configuration file is updated cyclically to roll back each operator in the configuration file one by one. The result processing module compares the running results on different computing devices, quantifies the difference and locates the abnormal operator. The operator test script generation module automatically generates reproducible test scripts based on the input and output parameters of the abnormal operator. The automatic model anomaly detection program is equivalent to the master control module, coordinating other modules to complete the automatic troubleshooting process of abnormal operators, calling the model operation module in sequence to generate results, triggering the operator list processing module to update the configuration, and looping back to perform verification. According to the difference analysis results of the result processing module, it is decided to continue the test or terminate the loop.

[0074] The module interaction and overall process are as follows: the model running module generates benchmark results and candidate results on the GPU and TPU respectively; the operator list processing module extracts the list of operators to be verified called by the TPU and generates a configuration file; the automatic troubleshooting program rolls back the operators in the configuration file to the CPU in turn, triggering the model running module to re-execute the target model; the result processing module compares the verification results with the benchmark results to determine the abnormal operator; if the abnormal operator is located, the test script generation module generates a debugging script and exits; if not located, the configuration file is updated and the next cycle is entered.

[0075] like Figure 3 ,The front end of the deep learning framework communicates with the back end scheduling layer through the interface module. The back end scheduling layer determines whether it is necessary to register the operator (such as operator 1, operator 2…operator n) based on the configuration file, and then determines whether the corresponding operator needs to be rolled back to the verification computing device.

[0076] like Figure 4As shown, a specific embodiment process is as follows: run the target model on the first computing device to generate a benchmark result; generate a list of operators to be verified according to the operators called during the operation of the target model; generate a configuration file according to the list of operators to be verified; run the target model, and roll back the operators in the configuration file to the verification computing device one by one; save the verification result; verify whether the result is consistent with the benchmark result; if not, re-enter the step of generating a configuration file according to the list of operators to be verified; if so, determine it as an abnormal operator, and automatically generate a test script corresponding to the abnormal operator according to the input and output parameter information and the string name, and re-enter the step of generating a configuration file according to the list of operators to be verified. It should be understood that the step of re-entering the step of generating a configuration file according to the list of operators to be verified is to verify the next operator to be verified in the list of operators to be verified.

[0077] Specifically, the specific implementation of this specific embodiment is as follows:

[0078] Step 1: Generate benchmark results and candidate results: Run the target model on the GPU (first computing device), input the preset problem, and save the output to llama_gpu.txt (the file corresponding to the benchmark result); run the same target model on the TPU (second computing device), enable the operator name recording function, save the output to llama_tpu.txt (the file corresponding to the candidate result), and generate the operator list to be verified opers.txt based on the difference between the candidate results and the benchmark results.

[0079] Step 2: Dynamically generate a YAML configuration file: read the operator list to be verified opers.txt, extract all operator names (such as add, mm, silu); create fallback_config.yml and mark the operators to be rolled back.

[0080] Step 3: Roll back operators one by one and verify: The scheduling layer reads fallback_config.yml and rolls back each operator to the CPU (verification computing device) in turn; the verification process is to roll back the add operator for the first time, modify the configuration file to only include add, run the model and save the results to llama_tpu_fallback_add.txt (the file corresponding to the verification result); compare llama_tpu_fallback_add.txt (the file corresponding to the verification result) with llama_gpu.txt (the file corresponding to the benchmark result). If the results are consistent, it means that add is implemented abnormally on the TPU. If the results are inconsistent, continue to roll back the next operator (such as mm); repeat until all operators in the list of operators to be verified are tested.

[0081] Step 4: Abnormal operator location and test script generation: Assume that the result after rolling back mm is consistent with the benchmark result, and determine that mm is an abnormal operator; enable the log function, save the input and output shape, step size and data type of the mm operator to mm_info.yaml (input and output parameter information of the abnormal operator), and automatically generate test_mm.py (test script) according to mm_info.yaml to reproduce the abnormality.

[0082] Among them, the scheduling layer and the operator registry, the registry structure uses std::map, the key is the string name of the operator (such as mm), and the value is a pointer to the adaptation layer function. The registry is filled through the RegisterPlugin function, skipping the operators marked in the configuration file, registering the TPU operator to the PyTorch scheduling layer, and the unregistered operator triggers CPU fallback, and the fallback function calls the PyTorch native CPU implementation.

[0083] The automated implementation process of this embodiment can quickly locate abnormal operators and directly lock the implementation defects of operators on the TPU through a single variable isolation strategy. The entire process from result comparison to test script generation is automated, reducing manual intervention.

[0084] This embodiment achieves efficient and accurate positioning of abnormal operators by dynamically controlling operator registration at the scheduling layer and managing fallback logic in the yaml configuration file, combined with result comparison and automatic script generation. It is particularly suitable for debugging scenarios when large models are deployed on heterogeneous hardware (such as GPU / TPU), significantly improving development efficiency and system stability.

[0085] In an exemplary embodiment, it also includes: obtaining the execution process of the abnormal operator in the target model according to the running result of the test script; analyzing the performance characteristics of the abnormal operator based on the execution process of the abnormal operator; adjusting the execution strategy of the abnormal operator according to the input parameters and computing conditions, and the adjustment strategy includes: dynamically adjusting the parallelism of the operator according to the size of the input data, the load of the hardware resources and the complexity of the computing task; dynamically adjusting the batch processing size of the operator according to the characteristics of the input data and the resource usage of the computing device; dynamically allocating or scheduling the hardware resources of the computing device according to factors such as the current load, memory usage, and bandwidth requirements of the computing device; using a machine learning algorithm to predict the optimal execution strategy based on historical operator execution data and performance indicators, and optimizing the execution strategy of the target operator in real time; automatically triggering an adjustment operation when an abnormality is detected in the execution process of the operator; tracking the actual effect of the adjusted execution strategy on the operator through a real-time monitoring system, and further optimizing the adjustment strategy based on feedback; and automatically recording the adjusted execution strategy for the execution process of the same operator in the future, so as to improve the execution efficiency of the operator and reduce the possibility of abnormality.

[0086] In this embodiment, by comprehensively analyzing the execution process of the abnormal operator, combining the performance characteristics and input and output parameters of the operator, and dynamically adjusting the parallelism, batch processing size and hardware resource allocation of the operator, it is possible to better adapt to different input data and computing conditions, optimize the execution efficiency of the operator, avoid excessive or insufficient resource allocation, and improve the operating performance of the overall model. Based on real-time monitoring and performance characteristic analysis, the execution strategy can be adjusted in time when an abnormality is detected, reducing the occurrence of abnormalities caused by hardware resource mismatch or unbalanced computing task load, thereby improving the stability of the model. The historical operator execution data and performance indicators are analyzed using machine learning algorithms to predict the optimal execution strategy of the operator and perform real-time optimization. This adaptive optimization mechanism can continuously improve the execution strategy as the usage time increases, so that the operator shows better computing power in different environments. Dynamically allocating resources according to factors such as the current load, memory usage, and bandwidth requirements of the computing device can effectively avoid resource bottlenecks, balance the computing requirements and resource usage of each operator, and thus reduce the overload and runtime errors of the computing device. The actual effect of the adjusted execution strategy on the operator can be tracked by the real-time monitoring system, and adjustments and optimizations can be made in time according to feedback. This makes the entire adjustment process more targeted and flexible, and can cope with changing computing needs. The adjusted execution strategy is automatically recorded so that it can be reused in the subsequent execution of the same operator. This not only provides experience and reference for the execution process of the same operator in the future, but also reduces the debugging and adjustment time at each execution, improving the overall efficiency and consistency of the system. By automating the adjustment strategy, the reliance on manual intervention is reduced. The detection, analysis and optimization of abnormal operators can be performed automatically, which improves the debugging efficiency and reduces the possibility of human error.

[0087] In general, this embodiment provides a more efficient and stable execution environment for abnormal operators by combining performance analysis, dynamic adjustment and intelligent optimization, which not only optimizes the execution efficiency of operators but also improves adaptability.

[0088] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method.

[0089] like Figure 5 As shown, an embodiment of the present application further provides an electronic device, including a memory 51 and a processor 52, wherein the memory 51 stores a computer program, and the processor 52 is configured to run the computer program to execute the steps in any of the above-mentioned abnormal operator positioning method embodiments.

[0090] For the description of the features in the embodiment corresponding to the electronic device, reference may be made to the relevant description of the embodiment corresponding to the method for locating an abnormal operator, which will not be described in detail here.

[0091] like Figure 6 As shown, an embodiment of the present application further provides a computer-readable storage medium 61, in which a computer program 62 is stored, wherein the computer program 62 is configured to execute the steps of any of the above-mentioned abnormal operator positioning method embodiments when running.

[0092] In an exemplary embodiment, the computer-readable storage medium 61 may include, but is not limited to, various media that can store the computer program 62, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0093] For the description of the features in the embodiment corresponding to the computer-readable storage medium 61, reference may be made to the relevant description of the embodiment corresponding to the method for locating an abnormal operator, which will not be described in detail here.

[0094] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any of the above-mentioned abnormal operator location method embodiments are implemented.

[0095] An embodiment of the present application further provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above-mentioned abnormal operator location method embodiments are implemented.

[0096] For the description of the features in the embodiment corresponding to the computer program product, reference may be made to the relevant description of the embodiment corresponding to the method for locating an abnormal operator, which will not be described in detail here.

[0097] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0098] The above is a detailed introduction to the method, device, storage medium and program product for locating an abnormal operator provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A method for locating an abnormal operator, characterized in that: include: Obtaining the running results of the target model on at least two different computing devices; When it is determined that an abnormal operator exists according to the difference of the running results, a list of operators to be verified is generated according to the operators called during the running of the target model; The abnormal operator is at least one operator called during the operation of the target model; Returning the operators in the list of operators to be verified to the verification computing device one by one, so that the verification computing device runs the target model according to the returned operators to obtain a verification result; According to the verification result and the operation result, the abnormal operator is located.

2. The method for locating an abnormal operator according to claim 1, characterized in that: Obtain the running results of the target model on at least two different computing devices, including: Running the target model on a first computing device to generate a benchmark result; Running the target model on a second computing device to generate candidate results; the running results include the benchmark results and the candidate results; Determining the existence of an abnormal operator according to the difference of the running results includes: Determine whether the difference between the candidate result and the benchmark result meets a preset difference condition; If satisfied, it is determined that an abnormal operator exists when the second computing device runs the target model.

3. The method for locating an abnormal operator according to claim 2, characterized in that: The first computing device is a reference computing device whose running results have been verified, and the second computing device is a computing device to be verified.

4. The method for locating an abnormal operator according to claim 2, characterized in that: Determining whether the difference between the candidate result and the benchmark result meets a preset difference condition includes: Determine whether the difference between the candidate result and the benchmark result is greater than a preset threshold; If so, it is determined that the preset difference condition is met.

5. The method for locating an abnormal operator according to claim 2, characterized in that: According to the verification result and the operation result, locating the abnormal operator includes: The abnormal operator is located according to the difference between the verification result and the benchmark result.

6. The method for locating an abnormal operator according to claim 5, characterized in that: Locating the abnormal operator according to the difference between the verification result and the benchmark result includes: Determining whether the verification result is the same as the benchmark result; If the verification result is the same as the benchmark result, the currently rolled back operator is determined to be the abnormal operator; If the verification result is different from the benchmark result, the currently rolled back operator is determined to be a normal operator.

7. The method for locating an abnormal operator according to claim 1, characterized in that: After generating a list of operators to be verified according to the operators called during the operation of the target model, the method further includes: Generate a configuration file according to the list of operators to be verified; Returning the operators in the list of operators to be verified to the verification computing device one by one includes: The configuration file is parsed, and the operators in the configuration file are rolled back to the verification computing device one by one.

8. The method for locating an abnormal operator according to claim 7, characterized in that: Parsing the configuration file and returning the operators in the configuration file to the verification computing device one by one includes: Parse the configuration file to obtain the string name of the operator in the configuration file; In the process of registering operators, registration of operators in the configuration file is skipped according to the string names of the operators in the configuration file, so that the operators in the configuration file are rolled back one by one to the verification computing device when the target model is run.

9. The method for locating an abnormal operator according to claim 8, characterized in that: Also includes: A predefined operator registry, which is used to store a mapping relationship between an operator string name and a corresponding execution logic; In the process of registering an operator, according to the string name of the operator in the configuration file, the registration of the operator in the configuration file is skipped, including: When the operator registration table is filled using the preset registration function, the filling of the operator corresponding to the target string name is skipped, and the operator registration table is filled according to other operators other than the operator in the configuration file; the target string name is the string name of the operator in the configuration file; When running the target model, the operators in the configuration file are rolled back to the verification computing device one by one, including: When running the target model, calling the populated operator registry; the populated operator registry does not include the operator in the configuration file; According to the populated operator registry, the operators in the configuration file are rolled back to the verification computing device one by one.

10. The method for locating an abnormal operator according to claim 9, characterized in that: After the predefined operator registry, it also includes: Define a configuration switch corresponding to the operator, the state of the configuration switch represents whether the corresponding operator is filled in the operator registration table, and the state of the configuration switch includes a closed state and an open state; When the operator registration table is filled using the preset registration function, the filling of the operator corresponding to the target string name is skipped, including: Setting the configuration switch corresponding to the target string name to the closed state; When the operator registration table is filled using the preset registration function, the filling of the operator corresponding to the closed state of the configuration switch is skipped, and the operator registration table is filled according to the operator whose state of the configuration switch is the open state.

11. The method for locating an abnormal operator according to claim 9, characterized in that: The operator registry is configured as a key-value pair structure, wherein the key is the string name of the operator and the value is the function corresponding to the execution logic of the operator.

12. The method for locating an abnormal operator according to any one of claims 1 to 11, characterized in that: Also includes: Record the input and output parameter information of the abnormal operator and the string name of the abnormal operator; The input and output parameter information includes input and output tensor shape, step size and data type; Generate a corresponding test script according to the input and output parameter information, the string name, and a preset script generation operator; The test script is used to reproduce the calculation process of the abnormal operator.

13. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the method for locating an abnormal operator according to any one of claims 1 to 12 when executing the computer program.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the method for locating an abnormal operator according to any one of claims 1 to 12.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method for locating an abnormal operator according to any one of claims 1 to 12 are implemented.

Images