Abnormal operator positioning method, device and system and related equipment

Abstract

The invention discloses an abnormal operator positioning method, device and system and related equipment, and relates to the technical field of artificial intelligence. The first computing node sends input data of the target operator to the second computing node in the Nth round of training process of the AI model, and the first computing node and the second computing node comprise different types of hardware. And the second computing node executes the target operator according to the input data to obtain a first execution result. And the first computing node continuously executes the Nth round of training process by using the first execution result, and when the deviation between the loss value of the AI model corresponding to the first execution result and the reference loss value meets a preset condition, the first computing node determines that the target operator is an abnormal operator influencing the loss value during AI model training. Therefore, whether the target operator has a great influence on the loss value of the AI model or not is determined by operating the target operator at the second computing node and observing the loss value of the AI model, so that the precision of positioning the operator influencing the loss value of the AI model can be improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an anomaly operator localization method, apparatus, system and related equipment. Background Technology

[0002] In real-world applications, there may be a need to migrate artificial intelligence (AI) models from hardware platform 1 to hardware platform 2. For example, AI models can typically be deployed on graphics processing units (GPUs), but when it is necessary to expand the hardware platforms on which AI models can be deployed, it is also possible to try deploying AI models on hardware platforms such as neural network processing units (NPUs).

[0003] Typically, when migrating AI models, users expect the loss values ​​produced by iterative training on two hardware platforms to remain similar. For example, a user might expect the loss value (used to calculate the gradients needed to update the parameters in the AI ​​model) from the Nth iteration of training on hardware platform 2 to deviate from the loss value from the Nth iteration on hardware platform 1 by no more than a threshold, where N is a positive integer. However, even with AI models executing the same code, differences in the underlying implementation logic across different hardware platforms can lead to significant differences in the loss values ​​produced by iterative training of the same AI model on different platforms, thus affecting the training of the AI ​​model on each hardware platform.

[0004] Currently, when a significant difference is found in the loss values ​​of an AI model during iterative training on two hardware platforms, users typically identify the anomalous operators that significantly impact the training loss from among the multiple operators included in the AI ​​model. In practice, for each operator in the AI ​​model, users usually deploy the operator on both hardware platform 1 and hardware platform 2, and then test the operator on each platform using test cases constructed specifically for that operator. Therefore, when the test results output by the operator based on the same test cases differ significantly across different hardware platforms, users can determine that the operator has a significant impact on the training loss value of the AI ​​model.

[0005] However, this method of identifying anomalous operators that affect the training loss value of an AI model has low accuracy. For example, although the computational results of some operators may differ significantly on different hardware platforms, these operators may be called less frequently during the entire training process. Alternatively, the different outputs of these operators may be smoothed out by the computation process of other operators (or functions) in the AI ​​model. This makes the actual impact of these operators on the training loss value of the AI ​​model small, but they may be incorrectly identified as anomalous operators that have a significant impact on the training loss value of the AI ​​model. Summary of the Invention

[0006] This application provides an anomaly operator localization method, aiming to improve the accuracy of locating anomaly operators that affect the training loss value of AI models. Furthermore, this application also provides anomaly operator localization apparatus, anomaly operator localization system, computing device, computer-readable storage medium, and computer program product.

[0007] Firstly, this application provides an anomaly operator localization method, which can be applied to an anomaly operator localization system. The system includes a first computing node and a second computing node, wherein the first and second computing nodes include different types of hardware, such as a GPU in the first computing node and an NPU in the second computing node. The hardware in the first computing node is used to run an AI (artificial intelligence) model, and the hardware in the second computing node is used to run the target operator in the AI ​​model. The target operator can be an operator executed during forward computation (i.e., a forward operator) or an operator executed during backward computation (i.e., a backward operator). The first computing node can be used to build a training environment, and the second computing node can be used to build a benchmark environment. The anomaly operator localization method includes: during the Nth round of training of the AI ​​model, the first computing node sends the input data of the target operator, where N is a positive integer, to the second computing node; the second computing node executes the target operator based on the input data, obtains a first execution result, and sends the first execution result back to the first computing node. Then, the first computing node continues to execute the Nth round of training process for the AI ​​model using the first execution result. When the deviation between the loss value of the AI ​​model corresponding to the first execution result and the reference loss value meets the preset condition, the first computing node determines the target operator as an abnormal operator that affects the loss value during AI model training.

[0008] During the training of the AI ​​model using the first computing node, the first computing node utilizes the second computing node to execute the target operator within the AI ​​model and uses the execution result of this target operator to continue training the AI ​​model. This allows verification of whether running the target operator on the second computing node causes a significant change in the AI ​​model's loss value, i.e., verifying whether the target operator has a significant impact on the AI ​​model's loss value during training. Thus, by running the target operator on the second computing node and observing the AI ​​model's loss value, we can determine whether the target operator has a significant impact on the AI ​​model's loss value, thereby improving the accuracy of locating operators affecting the AI ​​model's loss value. Furthermore, since locating operators affecting the AI ​​model's loss value during the training process uses real sample data required for training the AI ​​model, rather than constructed fake test cases, this ensures the accuracy of locating operators affecting the loss value. Additionally, since the second computing node does not need to run the entire AI model, but only the target operator within the AI ​​model, it is not required that the second computing node support running the entire AI model. This effectively reduces the hardware requirements for the second computing node in locating operators affecting the loss value.

[0009] In one possible implementation, the method further includes: a first computing node acquiring a configuration file, which includes an identifier for the target operator; and the first computing node deploying the target operator on a second computing node according to the configuration file. In this way, the target operator to be verified can be deployed on the second computing node based on the user's configuration for the operator, thereby verifying whether the user-configured target operator is an anomalous operator. Moreover, the second computing node only needs to run the target operator, without having to run the entire AI model, which effectively reduces the hardware requirements and resource consumption on the second computing node during the process of locating operators that affect the loss value.

[0010] In one possible implementation, the target operator is a forward operator, and the loss value of the AI ​​model corresponding to the first execution result is the loss value generated by the AI ​​model during the Nth round of training. Specifically, during the Nth round of training of the AI ​​model, when the first computing node sends the input data of the target operator to the second computing node, the first computing node may package the input data of the forward operator into a file during the forward computation of the Nth round of training and send the file to the second computing node. In this way, the first computing node can transmit input data to the second computing node by sending a packaged file, thus meeting the input data requirements of the second computing node for executing the target operator.

[0011] In one possible implementation, when the first computing node continues to execute the Nth round of training for the AI ​​model using the first execution result, it can specifically replace the value in the second execution result obtained by the first computing node executing the target operator with the value in the first execution result to obtain a third execution result, and then continue to execute the Nth round of training for the AI ​​model based on the third execution result. In this way, the first computing node can realize the forward replacement of the target operator and achieve non-intrusive modification of the machine learning framework used to train the AI ​​model.

[0012] In one possible implementation, the target operator is a backward operator, the first execution result is the gradient, and the loss value of the AI ​​model corresponding to the first execution result is the loss value generated by the AI ​​model during the (N+1)th round of training. Specifically, when the first computing node sends the input data of the target operator to the second computing node during the Nth round of training of the AI ​​model, it can do so during the backward computation of the Nth round of training, sending the accumulated gradient and the input data of the forward operator corresponding to the backward operator to the second computing node. Here, the accumulated gradient is the gradient calculated by the operator preceding the backward operator in the backpropagation sequence. Thus, based on the input data of the forward operator and the accumulated gradient corresponding to the previous backward operator, the gradient of the current backward operator is calculated. This improves the accuracy of gradient calculation, thereby helping to improve the accuracy of locating abnormal operators.

[0013] In one possible implementation, when the second computing node executes the target operator based on the input data to obtain the first execution result, it can specifically execute the forward operator based on the input data of the forward operator to obtain the output result, and then execute the backward operator based on the output result and the accumulated gradient to obtain the gradient (i.e., the first execution result). In this way, the second computing node can ensure the accuracy of gradient calculation by recalculating the output result of the forward operator and using it to calculate the gradient.

[0014] In one possible implementation, the anomaly operator localization system further includes a machine learning framework, which includes a gradient calculation function. Specifically, when the second computation node executes the inverse operator based on the output and accumulated gradient to obtain the gradient, it can execute the gradient calculation function based on the output and accumulated gradient to obtain the gradient. Thus, even when the machine learning framework used to train the AI ​​model does not support the inverse operator, the gradient calculation function can be used to calculate the gradient of the inverse operator, eliminating the need to develop an additional inverse operator within the machine learning framework and reducing the requirements on the machine learning framework.

[0015] In one possible implementation, the machine learning framework used to train the AI ​​model includes the PyTorch framework or the TensorFlow framework. In practice, other types of frameworks can also be used to train the AI ​​model.

[0016] In one possible implementation, when the hardware executing the target operator in the second computing node remains idle for a preset duration, the second computing node can also release the computing resources corresponding to the hardware executing the target operator. This avoids prolonged waste of computing resources in the second computing node and improves its resource utilization.

[0017] Secondly, this application provides an anomaly operator localization method, which is executed by a corresponding anomaly operator localization device. The method includes: the anomaly operator localization device instructing a first computing node to send input data of a target operator in the AI ​​model to a second computing node during the Nth round of training of an AI model, where N is a positive integer; the first computing node and the second computing node include different types of hardware, the hardware in the first computing node is used to run the AI ​​model, and the hardware in the second computing node is used to run the target operator; furthermore, the anomaly operator localization device instructs the second computing node to execute the target operator according to the input data, obtain a first execution result, and send the first execution result to the first computing node; then, the anomaly operator localization device instructs the first computing node to continue executing the Nth round of training for the AI ​​model using the first execution result; thereby, when the deviation between the loss value of the AI ​​model corresponding to the first execution result and the reference loss value meets a preset condition, the anomaly operator localization device determines that the target operator is an anomaly operator affecting the loss value during AI model training.

[0018] In one possible implementation, the anomaly operator locator acquires a configuration file, which includes the identifier of the target operator, and deploys the target operator on a second computing node according to the configuration file.

[0019] In one possible implementation, the target operator is a forward operator, and the loss value of the AI ​​model corresponding to the first execution result is the loss value generated by the AI ​​model during the Nth round of training. Then, when the abnormal operator localization device instructs the first computing node to send the input data of the target operator to the second computing node during the Nth round of training of the AI ​​model, it specifically instructs the first computing node to package the input data of the forward operator into a file during the forward calculation of the Nth round of training of the AI ​​model and send the file to the second computing node.

[0020] In one possible implementation, when the anomaly operator localization device instructs the first computing node to continue the Nth round of training for the AI ​​model using the first execution result, it may specifically instruct the first computing node to use the value in the first execution result to replace the value in the second execution result obtained by the first computing node in executing the target operator, thereby obtaining a third execution result, and instruct the first computing node to continue the Nth round of training for the AI ​​model based on the third execution result.

[0021] In one possible implementation, the target operator is the inverse operator, the first execution result is the gradient, and the loss value of the AI ​​model corresponding to the first execution result is the loss value generated by the AI ​​model in the N+1th round of training. Then, when the anomaly operator localization device instructs the first computing node to send the input data of the target operator to the second computing node during the Nth round of training of the AI ​​model, it can specifically instruct the first computing node to send the accumulated gradient and the input data of the forward operator corresponding to the inverse operator to the second computing node during the inverse calculation process of the Nth round of training of the AI ​​model. The accumulated gradient is the gradient calculated by the operator preceding the inverse operator in the backpropagation sequence.

[0022] In one possible implementation, when the anomaly operator localization device instructs the second computing node to execute the target operator based on the input data and obtain the first execution result, it may specifically instruct the second computing node to execute the forward operator based on the input data of the forward operator and obtain the output result, and instruct the second computing node to execute the reverse operator based on the output result and the accumulated gradient to obtain the gradient.

[0023] In one possible implementation, the anomaly operator localization system further includes a machine learning framework, which includes a gradient calculation function; then, when the anomaly operator localization device instructs the second computing node to execute the inverse operator based on the output result and the accumulated gradient to obtain the gradient, it may specifically instruct the second computing node to execute the gradient calculation function based on the output result and the accumulated gradient to obtain the gradient.

[0024] In one possible implementation, the machine learning framework includes the PyTorch framework or the TensorFlow framework.

[0025] The anomaly operator localization method provided in the second aspect corresponds to the anomaly operator localization method provided in the first aspect. Therefore, the technical effects of the second aspect and any implementation thereof can be found in the relevant descriptions of the technical effects of the first aspect and its corresponding implementation thereof, and will not be repeated here.

[0026] Thirdly, this application provides an anomaly operator localization device, comprising: an instruction module, configured to instruct a first computing node to send input data of a target operator in the AI ​​model to a second computing node during the Nth round of training of an AI model, where N is a positive integer; the first computing node and the second computing node include different types of hardware, the hardware in the first computing node is used to run the AI ​​model, and the hardware in the second computing node is used to run the target operator; instructing the second computing node to execute the target operator according to the input data, obtain a first execution result, and send the first execution result to the first computing node; instructing the first computing node to continue executing the Nth round of training for the AI ​​model using the first execution result; and a determination module, configured to determine that the target operator is an anomaly operator affecting the loss value during AI model training when the deviation between the loss value of the AI ​​model corresponding to the first execution result and the reference loss value meets a preset condition.

[0027] In one possible implementation, the anomaly operator localization device further includes: an acquisition module for acquiring a configuration file, the configuration file including the identifier of the target operator; and a deployment module for deploying the target operator on a second computing node according to the configuration file.

[0028] In one possible implementation, the target operator is a forward operator, and the loss value of the AI ​​model corresponding to the first execution result is the loss value generated by the AI ​​model during the Nth round of training; the instruction module is specifically used to: instruct the first computing node to package the input data of the forward operator into a file during the forward computation of the AI ​​model during the Nth round of training, and send the file to the second computing node.

[0029] In one possible implementation, the instruction module is specifically used to: instruct the first computing node to replace the value in the second execution result obtained by the first computing node in executing the target operator with the value in the first execution result to obtain the third execution result; and instruct the first computing node to continue to execute the Nth round of training process for the AI ​​model based on the third execution result.

[0030] In one possible implementation, the target operator is the inverse operator, the first execution result is the gradient, and the loss value of the AI ​​model corresponding to the first execution result is the loss value generated by the AI ​​model in the N+1th round of training; the instruction module is specifically used to: instruct the first computing node to send the accumulated gradient and the input data of the forward operator corresponding to the inverse operator to the second computing node during the inverse calculation of the Nth round of training of the AI ​​model, wherein the accumulated gradient is the gradient calculated by the operator preceding the inverse operator in the backpropagation sequence.

[0031] In one possible implementation, the instruction module is specifically used to: instruct the second computing node to execute the forward operator based on the input data of the forward operator and obtain the output result; and instruct the second computing node to execute the backward operator based on the output result and the accumulated gradient to obtain the gradient.

[0032] In one possible implementation, the anomaly operator localization system further includes a machine learning framework, which includes a gradient calculation function; and an instruction module, specifically used to instruct the second computing node to execute the gradient calculation function based on the output result and the accumulated gradient to obtain the gradient.

[0033] In one possible implementation, the machine learning framework includes the PyTorch framework or the TensorFlow framework.

[0034] The anomaly operator location device provided in the third aspect corresponds to the anomaly operator location method provided in the first aspect. Therefore, the technical effects of the third aspect and any implementation thereof can be found in the relevant descriptions of the technical effects of the first aspect and its corresponding implementation thereof, and will not be repeated here.

[0035] Fourthly, this application provides a computing device including a processor and a memory; wherein the memory is used to store instructions, and the processor executes the instructions stored in the memory to perform the operation steps of the anomaly operator localization method described in the second aspect or any implementation thereof.

[0036] Fifthly, this application provides an anomaly operator localization system, characterized in that the anomaly operator localization system includes a first computing node and a second computing node, the first computing node and the second computing node include different types of hardware, the hardware in the first computing node is used to run an artificial intelligence (AI) model, and the hardware in the second computing node is used to run the target operator in the AI ​​model; the first computing node is used to execute the operation steps performed by the first computing node in the first aspect or any implementation thereof, and the second computing node is used to execute the operation steps performed by the second computing node in the first aspect or any implementation thereof.

[0037] Sixthly, this application provides a computer-readable storage medium storing instructions that, when executed on a computing device, cause the computing device to perform the operation steps of the anomaly operator localization method described in the first aspect or any implementation thereof, or cause the computing device to perform the operation steps of the anomaly operator localization method described in the second aspect or any implementation thereof.

[0038] In a seventh aspect, this application provides a computer program product containing instructions that, when run on a computing device, causes the computing device to perform the operation steps of the anomaly operator location method described in the first aspect or any implementation thereof, or causes the computing device to perform the operation steps of the anomaly operator location method described in the second aspect or any implementation thereof.

[0039] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0040] Figure 1a A schematic diagram of the structure of an exemplary anomaly operator localization system provided in this application;

[0041] Figure 1b A diagram illustrating how a compute node calls the API provided by the Machine Learning Framework 200 to execute operators in an AI model.

[0042] Figure 2 A flowchart illustrating an anomaly operator localization method provided in this application;

[0043] Figure 3 A schematic diagram of an exemplary user interface provided for this application;

[0044] Figure 4 A schematic diagram of the interface design for the API used for forward replacement provided in this application;

[0045] Figure 5 A schematic diagram illustrating the logic of forward substitution provided in this application;

[0046] Figure 6 A schematic diagram of the interface design for the API used for reverse replacement provided in this application;

[0047] Figure 7 A schematic diagram illustrating the logic for implementing reverse substitution provided in this application;

[0048] Figure 8 A logical diagram illustrating the calculation of gradients using the forward and backward functions in the defined API provided in this application;

[0049] Figure 9 This is a logical diagram illustrating the calculation of gradients on computing node 102 provided in this application.

[0050] Figure 10 This application provides a schematic diagram of the structure of an anomaly operator localization device.

[0051] Figure 11 This is a schematic diagram of the hardware structure of a computing device provided in this application. Detailed Implementation

[0052] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, various non-limiting embodiments of the present application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained based on the embodiments in this application and based on the above content are within the scope of protection of this application.

[0053] Below, we will first introduce the AI ​​models and operators mentioned in this application.

[0054] An AI model is a mathematical model based on artificial intelligence technology that can automatically process and analyze input data and output corresponding results. For example, an AI model can output detection results for targets in an input image. Exemplary examples of AI models include language models (LM), generative pre-trained transformer (GPT) models, bidirectional encoder representations from transformer (BERT) models, vision transformer (ViT) models, contrastive language-image pre-training (CLIP) models, and other types of AI models; there is no limitation on this.

[0055] An AI model can include multiple operators, each of which is a basic unit that performs a specific mathematical or logical operation, such as a convolution operator for implementing convolution operations. During iterative training of the AI ​​model, operators executed in the forward computation process are called forward operators; operators executed in the backward computation process are called backward operators.

[0056] In practical applications, there may be a need to migrate the AI ​​model to be trained between different types of hardware environments. For example, in scenarios where the hardware environment for training AI models is expanded, some types of AI models are commonly trained on GPU-based hardware environments. In such cases, the AI ​​model can be migrated from a GPU-based hardware environment to an NPU-based hardware environment for training, thereby enriching the hardware environments available for training the AI ​​model. The hardware environment before the migration can be called the benchmark environment (or test baseline environment), and the hardware environment after the migration can be called the training environment.

[0057] Because benchmark and training environments are implemented on different types of hardware, the execution results of the same operator in an AI model may differ significantly depending on the hardware environment. This leads to a substantial difference between the loss value generated by iterative training in the benchmark environment and the loss value generated by iterative training in the training environment. Therefore, in practical applications, when expanding the hardware environment for training AI models, it is common practice to test whether there is a significant difference between the loss value generated by iterative training of the same AI model in the benchmark environment and the loss value generated by iterative training in the training environment. Furthermore, when a significant difference is found between the loss values ​​generated by iterative training of the same AI model in two different training environments, users usually identify the anomalous operator in the AI ​​model; this anomalous operator is the one that causes the large difference in loss values ​​during iterative training of the AI ​​model in different hardware environments.

[0058] However, if test cases identify operators whose execution results differ significantly across different hardware environments, and these operators are further classified as anomalous operators, the accuracy of locating anomalous operators will be low. This is because, during the iterative training of the AI ​​model, although the execution results of these operators may differ significantly across different hardware environments, these operators are called infrequently, or the various significantly different execution results of these operators may be smoothed into the same result by other operators in the AI ​​model (such as the softmax operator). Consequently, these operators have a relatively small impact on the loss value generated by the AI ​​model in each training round.

[0059] Therefore, this application provides Figure 1a An anomaly operator localization system 10 is provided, and the accuracy of identifying anomaly operators in the AI ​​model is improved in the anomaly operator localization system 10.

[0060] See Figure 1a A schematic diagram of an anomaly operator localization system is shown. Figure 1aAs shown, the anomaly operator localization system 10 includes multiple computing nodes. These nodes can communicate with each other via a network, such as through the Hypertext Transfer Protocol (HTTP). For ease of understanding and description, Figure 1a The following description uses computing nodes 101 and 102 as examples. Furthermore, the anomaly operator localization system 10 can also provide a client to external users (…). Figure 1a (Not shown in the image), this client can be used to interact with the user, such as obtaining configuration files provided by the user for operators in the AI ​​model. For example, the client could be a web browser or program code running on the user's terminal device.

[0061] Each computing node may include hardware that supports the execution of AI models or operators. For example, a computing node may be a computing server, a general-purpose processor, or an accelerator card, capable of supporting the execution of AI models or operators within AI models. Specifically, when a computing node is an accelerator card, it may be a deep-learning processing unit (DPU), GPU, NPU, or other types of accelerator cards. A computing node can also be a server containing one or more accelerator cards, and different computing nodes may include different types of accelerator cards. In this case, the computing node can utilize hardware such as accelerator cards to support the execution of AI models or operators. For example, computing node 101 can run an AI model via an NPU, while computing node 102 can run some operators within the AI ​​model via a GPU. The number of hardware components included in different computing nodes may also vary; for example, computing node 101 may include a larger number of accelerator cards to support the execution of a complete AI model, while computing node 102 may include a smaller number of accelerator cards, sufficient to support the execution of some operators within the AI ​​model.

[0062] A machine learning framework 200 can run on multiple computing nodes. This machine learning framework 200 can be configured with code files for multiple operators, such as... Figure 1a The operators shown are 1 to M (where M is a positive integer). Furthermore, the machine learning framework 200 provides application programming interfaces (APIs) corresponding to each operator. Thus, during the iterative training of the AI ​​model on a computing node (such as computing node 101 or computing node 102), when it is necessary to execute various operators in the AI ​​model, the computing node can call the APIs provided by the machine learning framework 200 to execute the code file of that operator, thereby executing the operator. For example, as... Figure 1bAs shown, taking the execution of operators A and B in the AI ​​model as an example, during the execution of operator A, the computing node calls the API corresponding to operator A provided by the machine learning framework 200. The input parameters of this API include the input data required to execute operator A. Then, the computing node can execute the program code in the code file of operator A, perform corresponding calculations based on the input data, and obtain the corresponding calculation result 1. Next, the computing node can call the API of operator B based on the calculation result 1. The input parameters of this API include the calculation result 1 (i.e., the input data required to execute operator B), so the computing node can execute the program code in the code file of operator B, perform calculations based on the calculation result 1, and obtain the calculation result 2. In this way, the computing node implements the execution of operators A and B.

[0063] For example, the machine learning framework 200 can be, for instance, the PyTorch framework or the TensorFlow framework, or other types of frameworks. When the PyTorch framework is deployed on multiple computing nodes, it can also be configured with a function for automatically calculating gradients for the parameters in the AI ​​model, hereinafter referred to as the gradient calculation function, so that the gradients of the parameters can be calculated during the backpropagation process of training the AI ​​model.

[0064] exist Figure 1a In the anomaly operator localization system 10 shown, some computing nodes can form a training environment, and some computing nodes can form a benchmark environment. For ease of description, the following explanation will use computing node 101 forming the training environment and computing node 102 forming the benchmark environment as an example.

[0065] Based on this, Figure 1a The anomaly operator localization system 10 shown includes an anomaly operator localization device 201. This device 201 can be implemented in software and runs within the machine learning framework 200. For example, it can be a plugin or software development kit (SDK) running within the machine learning framework 200. Furthermore, the anomaly operator localization device 201 can locate anomaly operators in the AI ​​model, that is, locate operators that affect the loss value of the AI ​​model. It should be noted that, for ease of description and understanding, the anomaly operators affecting the loss value of the AI ​​model described in this application refer to operators that have a relatively large impact on the loss value of the AI ​​model, such as operators whose execution results cause the change in the loss value to reach a threshold.

[0066] In specific implementation, the anomaly operator localization device 201 can acquire the AI ​​model, such as by downloading the AI ​​model file from the network, and deploy the AI ​​model to the computing node 101 in the training environment. Simultaneously, the anomaly operator localization device 201 can also obtain a user-provided configuration file through a client. This configuration file indicates one or more operators in the AI ​​model, hereinafter referred to as target operators. The target operators indicated in the configuration file are those operators that may affect the loss value of the AI ​​model.

[0067] The anomaly operator localization device 201 can instruct computing node 101 to iteratively train the AI ​​model. During the Nth (N is a positive integer) round of training of the AI ​​model, it deploys the target operator indicated in the configuration file on computing node 102 in the benchmark environment (or deploys the target operator to computing node 102 in advance), and instructs computing node 101 to send the input data of the target operator to computing node 102. Then, the anomaly operator localization device 201 can instruct computing node 102 to execute the target operator according to the input data, obtain the corresponding execution result, and send the execution result to computing node 101. Next, the anomaly operator localization device 201 instructs computing node 101 to continue the Nth round of training for the AI ​​model using the received execution result. Furthermore, the anomaly operator localization device 201 can compare the loss value of the AI ​​model calculated based on the execution result corresponding to the target operator with the reference loss value. The target operator can be the forward operator in the forward computation process of the Nth training round. In this case, the anomaly operator localization device 201 can compare the loss value calculated based on the execution result with the reference loss value corresponding to the Nth training round. Alternatively, the target operator can be the reverse operator in the reverse computation process of the Nth training round. In this case, the loss value calculated based on the execution result is the loss value at the (N+1)th training round, and the anomaly operator localization device 201 can compare the loss value at the (N+1)th training round with the reference loss value corresponding to the (N+1)th training round.

[0068] When the deviation between the calculated loss value of the AI ​​model and the reference loss value meets a preset condition (the deviation between the loss value and the reference loss value is greater than a threshold), the anomaly operator localization device 201 can determine that the target operator is an anomaly operator affecting the loss value during AI model training. Furthermore, the anomaly operator localization device 201 can also output a corresponding localization result, which indicates that the target operator is an anomaly operator. For example, the anomaly operator localization device 201 can output this localization result to the client so that the client can notify the user of the anomaly operator. Conversely, when the deviation between the calculated loss value of the AI ​​model and the reference loss value does not meet a preset condition, the anomaly operator localization device 201 can determine that the target operator does not affect the loss value during AI model training.

[0069] Thus, the anomaly operator localization device 201 determines whether a target operator affects the loss value during AI model training by running the target operator on the computing node 102 and observing the loss value of the AI ​​model. This allows direct observation of the relationship between the target operator running in different hardware environments and the loss value during AI model training, thereby improving the accuracy of anomaly operator localization. Moreover, since the target operator affecting the AI ​​model's loss value is located during the AI ​​model training process, the data used is the real sample data required for training the AI ​​model, rather than constructed fake test cases, which ensures the accuracy of anomaly operator localization.

[0070] In addition, during the process of locating anomalous operators in the AI ​​model, since the computing node 102 in the benchmark environment does not need to run the entire AI model, but only the target operator, it is not required that the computing node 102 can support running the entire AI model. This can effectively reduce the hardware requirements of the benchmark environment during the process of locating the question-and-answer problem operator.

[0071] It is worth noting that, Figure 1a The anomaly operator localization system 10 shown is only an example. In actual applications, the anomaly operator localization system 10 may also adopt other possible structures. The following is an example description.

[0072] In the first implementation example, Figure 1a In the anomaly operator localization system 10 shown, the anomaly operator localization device 201 is implemented in software as an example for illustration. In other possible anomaly operator localization systems, the anomaly operator localization device 201 can also be implemented in hardware, so that the machine learning framework 200 can locate the anomaly operators in the AI ​​model by calling the anomaly operator localization device 201.

[0073] For example, the anomaly operator localization device 201 can be implemented by a computing device including a processor, wherein the processor may be a central processing unit (CPU), or may be implemented as an application-specific integrated circuit (ASIC), or a programmable logic device (PLD), etc. The aforementioned PLD may be implemented as a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a digital processing unit (DPU), an electronic processing unit (NPU), a system-on-chip (SoC), an offload card, an accelerator card, or any combination thereof.

[0074] In the second implementation example, Figure 1a The anomaly operator localization system 10 shown is illustrated using an example where both the training and benchmark environments each include one computation node. In other possible anomaly operator localization systems, the training or benchmark environment may include two or more computation nodes.

[0075] In the third implementation example, in other possible anomaly operator localization systems, including Figure 1a Based on the multiple computing nodes shown, more types of devices can be included to support other functions of the anomaly operator localization system. For example, the anomaly operator localization system can also include a management node for managing computing nodes 101 and 102, etc.

[0076] For ease of understanding, the following description, in conjunction with the accompanying drawings, illustrates an embodiment of the anomaly operator for determining the loss value in an AI model provided in this application.

[0077] See Figure 2 , Figure 2 This is a flowchart illustrating an exemplary operator determination method provided in an embodiment of this application. Figure 2 The operator determination method shown can be applied to Figure 1a The anomaly operator localization system 10 shown can be applied to other possible anomaly operator localization systems. For ease of understanding and description, the following description uses an application... Figure 1a The anomaly operator localization system 10 shown is illustrated by way of example. The computing nodes 101 and 102 include different types of hardware for simulating the operation of operators in the AI ​​model on different types of hardware platforms. For example... Figure 2 As shown, the operator determination method may specifically include the following steps.

[0078] S201: The anomaly operator localization device 201 acquires the AI ​​model to be trained and the configuration file, which includes the identifier of the target operator in the AI ​​model.

[0079] In this embodiment, the client can provide a user interface (UI) to the user, and can prompt the user on the UI to locate the AI ​​model of the anomalous operator, so that the user can provide the corresponding AI model on the UI. For example, the client can present... Figure 3 The UI shown allows users to enter the Uniform Resource Locator (URL) of the AI ​​model in a text box and click the "Import" control. Accordingly, the client can obtain the AI ​​model from the network based on the URL and provide the AI ​​model to the anomaly operator locator 201 so that the anomaly operator locator 201 can obtain the AI ​​model.

[0080] Typically, only a few operators in an AI model may affect the loss value during training. Therefore, users can configure the operators to be verified in the AI ​​model (i.e., verify whether the operator is an anomalous operator affecting the loss value) on the UI interface. Specifically, the UI interface can display something like this: Figure 3 The system displays multiple candidate operators, which the user can select. The client then identifies the selected operator as the target operator to be verified and generates a configuration file containing the identifier of that target operator. Alternatively, the user can manually develop a configuration file that specifies the operators in the AI ​​model that need to be verified. In this case, the user can enter the file path of the configuration file in a text box on the UI interface and import the user-developed configuration file into the client by clicking the "Import" control. The client can then send the configuration file to the abnormal operator location device 201.

[0081] The user-specified operators to be verified can be either forward operators in the AI ​​model (operators executed during the forward computation of the AI ​​model) or backward operators (operators executed during the backward computation of the AI ​​model). Alternatively, the user can specify operators to be verified by including either forward or backward operators. When the configuration file includes identifiers for multiple operators, these identifiers can be separated by symbols (such as commas).

[0082] In practical applications, users can iteratively train the same AI model in both a training environment and a benchmark environment. These training and benchmark environments can be configured through... Figure 1a The anomaly operator localization system 10 shown can be constructed using computing nodes, or it can be constructed using computing nodes in other systems / clusters. Furthermore, during the iterative training of the AI ​​model, both the training environment and the benchmark environment can generate logs to record the loss value of the AI ​​model in each round of training iteration. After X (X is a positive integer) rounds of training, the user can compare the differences between the loss values ​​generated by the AI ​​model in the same round of training in the training environment and the benchmark environment based on the logs in the training environment and the benchmark environment. Moreover, when the loss values ​​differ significantly, the user can further deploy the same operator (the operator in the AI ​​model) in both the training environment and the benchmark environment, and execute the operator in both environments using test cases (i.e., calling the API provided by the operator using test cases). Then, when there is a significant difference between the execution results of the operator in the training environment and the execution results in the benchmark environment based on the test cases, the user can determine that the operator is an anomaly operator that may affect the loss value. Using a similar approach, users can verify multiple operators in an AI model one by one, thereby filtering out operators that may affect the loss value from the multiple operators included in the AI ​​model.

[0083] It is understandable that operators whose execution results vary significantly under different hardware environments may affect the loss value of the AI ​​model, or they may not affect the loss value of the AI ​​model due to insufficient calls or interference from other operators. Therefore, in this embodiment, these operators can be further verified by the abnormal operator localization system 10 to determine whether these operators actually affect the loss value of the AI ​​model. Users can then use the filtering results to... Figure 3 The UI shows the operators that need to be verified. The user can specify the operators to be verified, which can be all the filtered operators or only some of the operators.

[0084] It is understood that the above-described implementation method of the anomaly operator localization device 201 acquiring the AI ​​model and configuration file is only illustrative. In other embodiments, the anomaly operator localization device 201 may also acquire the AI ​​model and configuration file through other means.

[0085] Furthermore, the configuration file acquired by the anomaly operator locator 201 can configure not only multiple user-specified operators, but also other types of information. For example, the configuration file can also indicate the computing nodes used to participate in verifying anomaly operators, such as the IP addresses or port numbers of computing node 101 and computing node 102. As another example, the configuration file can also configure logs used to record information such as the operation of each operator during the training of the AI ​​model, such as configuring the log output path and the log level. The log levels can support "error," "info," and "debug" levels.

[0086] S202: The anomaly operator localization device 201 instructs the computing node 101 to send the input data of the target operator in the AI ​​model to the computing node 102 during the Nth round of training of the AI ​​model.

[0087] S203: The anomaly operator location device 201 instructs the computing node 102 to execute the target operator according to the input data, obtain execution result 1, and send execution result 1 to the computing node 101.

[0088] S204: The anomaly operator localization device 201 instructs the computing node 101 to continue the Nth round of training for the AI ​​model using the execution result 1.

[0089] Where N is a positive integer.

[0090] In this embodiment, the anomaly operator localization device 201 can verify whether the target operator in the AI ​​model is an anomalous operator during the training process. To this end, the anomaly operator localization device 201 can deploy the AI ​​model to the computing node 101 and instruct the computing node 101 to run the AI ​​model, that is, deploy the AI ​​model to the training environment for operation. Then, the anomaly operator localization device 201 can instruct the computing node 101 to begin iterative training of the AI ​​model using training samples.

[0091] During one round of AI model training, computation node 101 can first perform a forward computation process on the AI ​​model using the input data from the training samples. Specifically, this can involve sequentially executing multiple forward operators in the AI ​​model based on the training samples to obtain the output result (i.e., the inference result) of the AI ​​model. Then, computation node 101 can calculate the difference between the output result and the label in the training samples (i.e., the correct output result) to obtain a loss value. The larger the loss value, the greater the difference between the output result and the label. Next, computation node 101 can perform a backward computation process on the AI ​​model based on the loss value. Specifically, this can involve sequentially executing multiple backward operators in the AI ​​model and using the gradients calculated by each backward operator to update the parameters in the forward operators of the AI ​​model. Each backward operator can correspond to at least one forward operator, and the gradient calculated by the backward operator is used to update the parameters in its corresponding forward operator.

[0092] During forward computation, the output of the AI ​​model is obtained by progressively calculating the results of executing multiple forward operators. Therefore, the execution results of the forward operators may affect the output of the AI ​​model, thereby indirectly affecting the loss value of the AI ​​model. That is, the target operator specified by the user for verification can be one or more forward operators that need to be executed during the forward computation process.

[0093] During the backward computation process, after executing each backward operator based on the loss value, the parameters in the forward operators are updated according to the calculated gradient. These updated parameters then affect the execution results of the forward operators in the next round of forward computation, indirectly influencing the loss value of the AI ​​model. Therefore, the target operator specified by the user for verification can also be one or more forward operators required to be executed during the backward computation process.

[0094] Typically, the machine learning framework 200 can include the program code for all operators (including forward and backward operators) in the AI ​​model and provide APIs for calling each operator. Thus, in the forward computation phase, the computing node 101 can execute each forward operator in the AI ​​model by calling the API corresponding to each forward operator; in the backward computation phase, the computing node 101 can execute each backward operator in the AI ​​model by calling the API corresponding to each backward operator. In practical applications, the AI ​​model can also include user-defined operators, and the AI ​​model file includes the code file for the user-defined operator. Therefore, the computing node 101 can directly execute the code file to execute the user-defined operator (without needing to call the APIs in the machine learning framework 200). For ease of understanding, the following explanation uses the example of the computing node executing operators in the AI ​​model by calling APIs.

[0095] Below, taking the Nth round of training of the AI ​​model as an example, we will introduce the Nth round of training process when the target operator is the forward operator and the Nth round of training process when the target operator is the backward operator.

[0096] In scenario one, the target operator is specifically a forward operator, and there is only one target operator. Then, during the forward computation process of the Nth round of training, computation node 101 can sequentially call the APIs corresponding to the forward operators in the machine learning framework 200 according to the computation graph corresponding to the forward computation process, to execute each forward operator in the AI ​​model. Here, a computation graph is a graph structure used to describe the data computation logic in an AI model. Typically, a computation graph consists of nodes and edges, where nodes represent operations (e.g., addition, multiplication, activation functions, etc.), and edges represent data flow (i.e., data input and data output).

[0097] When the execution reaches the target operator (forward operator) indicated in the configuration file, the anomaly operator locator 201 can instruct computing node 101 to send the input data required for executing the target operator to computing node 102. Furthermore, the anomaly operator locator 201 can also instruct computing node 101 to send the identifier of the API required for executing the target operator to computing node 102. In this way, computing node 102 can call the API indicated by the identifier and execute the target operator on computing node 102 according to the received input data, obtaining the execution result 1 of the target operator on computing node 102. Then, computing node 102 can feed back the execution result 1 to computing node 101. Accordingly, computing node 101 can use the execution result 1 to continue executing subsequent forward and backward computation processes, thereby completing the Nth round of training for the AI ​​model.

[0098] In practical applications, computation node 101 can record the execution result of each forward operator and the gradient attributes associated with the computational operations in the forward operator (such as grad_fn in the PyTorch framework) in the computation graph. These gradient attributes can be used to calculate the gradient for updating parameters of the forward operator during the backward computation process. In some machine learning frameworks 200, if computation node 101 directly executes subsequent forward computation processes (and backward computation processes) based on the execution result 1 generated by computation node 102, it is easy for the gradient attributes associated with the computational operations in the target operator to be lost in the computation graph, thus affecting the process of calculating the gradient for the target operator during the backward computation process. Therefore, computation node 101 still executes the target operator during the forward computation process, obtaining execution result 2. Then, the abnormal operator localization device 201 can instruct computation node 101 to replace the value in execution result 2 generated by computation node 101 with the value (data) in execution result 1 to obtain execution result 3 (without changing other attributes in the execution result), and continue the Nth round of training for the AI ​​model based on execution result 3. Typically, the execution result obtained from executing the target operator can include not only numerical values, but also information such as shape, data type (dtype), and value range. Thus, computation node 101 can continue to record gradient attributes in the computation graph to ensure gradient calculation during the backpropagation process. Simultaneously, computation node 101 continues the Nth round of training based on the numerical values ​​generated by running the target operator on computation node 102, in order to subsequently verify whether the target operator is an anomalous operator.

[0099] As an implementation example, a persistent service can run on compute node 102, allowing compute node 102 to respond to and execute processes targeting a target operator. Specifically, before training the AI ​​model, this persistent service can be configured on compute node 102 by the anomaly operator locator 201, which utilizes the hardware resources on compute node 102 to start and run the persistent service. The specific implementation process for enabling and disabling the persistent service on compute node 102 is described below. Furthermore, the persistent service can provide an API for forward substitution, which compute node 101 can call to execute the target operator on compute node 102 and return the corresponding execution result 1 for that target operator.

[0100] The API provided by the persistence service can be, for example, using... Figure 4 The interface design shown includes the name of the API corresponding to the forward operator executed on the computing node 102 and the input data required to execute the forward operator. The result returned by the API call is a file including the execution result 1.

[0101] Then, when computation node 101 executes the target operator, such as Figure 5 As shown, computing node 101 can call, based on the input data of the target operator and the name of the API corresponding to the target operator, such as... Figure 4 The API named "run-api-result" shown is used for forward substitution to request execution result 1. The exception operator locator 201 can pre-wrap the API corresponding to the target operator using a decorator, so that when the computing node 101 calls the API corresponding to the target operator, it can invoke... Figure 4 The APIs shown for forward replacement, such as the exception operator locator 201, can be added to the API corresponding to the target operator. Figure 4 The program code for the API shown.

[0102] For example, during the invocation of the API for forward replacement, compute node 101 can copy the input data required to execute the target operator into memory, and package the input data into a pickle-formatted (or other formatted) file in memory. Then, compute node 101 can send an HTTP request including the file and the API name corresponding to the target operator to invoke the API for forward replacement. Figure 5 As shown. Accordingly, assuming that compute node 102 uses the GPU to execute the target operator, after the API is called, the persistence service on compute node 102 can cache the input data in the request to the GPU memory, and call the API corresponding to the target operator in the machine learning framework 200 according to the API name carried in the request, and execute the target operator corresponding to the API based on the input data in the GPU memory, to obtain the corresponding execution result 1, such as... Figure 5 As shown, execution result 1 can be cached in GPU memory. During the process of feeding execution result 1 back to compute node 101, the persistence service can copy execution result 1 from GPU memory to the memory of compute node 102, and package execution result 1 into a pickle format (or other format) file in memory. This allows the service to send an HTTP request to compute node 101, thus feeding back execution result 1. Compute node 101 can then use the values ​​in execution result 1 to replace the values ​​in execution result 2 generated by the target operator, thereby completing the forward replacement of the target operator and achieving non-intrusive modification of the machine learning framework 200.

[0103] In scenario two, the target operator is specifically a reverse operator, and there is only one reverse operator. During the reverse computation of the Nth training round, computation node 101 can execute each reverse operator in the AI ​​model sequentially, from the output layer to the input layer, to update the parameters in the corresponding forward operator using the gradients obtained from executing the reverse operators. When the target operator (reverse operator) indicated in the configuration file is executed, the abnormal operator location device 201 can instruct computation node 101 to send the data required to execute the target operator and the identifier of the target operator's API to computation node 102. In this way, computation node 102 can call the API indicated by the identifier and execute the target operator based on the data, obtaining the corresponding execution result 1. At this time, execution result 1 can specifically be the gradient used to update the parameters in the forward operator corresponding to the reverse operator.

[0104] As an implementation example, a persistent service can run on compute node 102, allowing compute node 102 to respond to and execute processes targeting a target operator. Specifically, before training the AI ​​model, this persistent service can be configured on compute node 102 by the anomaly operator localization device 201, which utilizes the hardware resources on compute node 102 to start and run the persistent service. Furthermore, the persistent service can provide an API for reverse substitution, which compute node 101 can call to execute the target operator on compute node 102 and return the execution result 1 (for updating the gradient of parameters) corresponding to the target operator.

[0105] In practical applications, some types of machine learning frameworks can provide APIs for inverse operators, while other types of machine learning frameworks may not provide APIs for inverse operators.

[0106] When the machine learning framework 200 provides an API for inverse operators, the API provided by the persistence service for inverse replacement, for example, could be... Figure 6 The interface design shown includes the name of the API corresponding to the inverse operator executed on computing node 102 and the input data required to execute the inverse operator. The result returned by the API call is a file including the execution result 1.

[0107] Therefore, when compute node 101 executes the target operator, compute node 101 can call, based on the input data of the target operator and the name of the API corresponding to the target operator, such as... Figure 6The API named "api-autograd" shown is used for reverse replacement to request execution result 1. The exception operator locator 201 can pre-wrap the API corresponding to the target operator using a decorator, so that when the computing node 101 calls the API corresponding to the target operator, it can call... Figure 6 The API shown for reverse replacement, such as the exception operator locator 201, can be added to the API corresponding to the target operator. Figure 6 The program code for the API shown.

[0108] Accordingly, after the API for reverse replacement is invoked, the persistence service on compute node 102 can invoke the API of the reverse operator indicated by the identifier, execute the reverse operator according to the input data, obtain the corresponding execution result 1, and feed back the execution result 1 to compute node 101, thereby realizing the reverse replacement of the target operator.

[0109] When the machine learning framework 200 does not provide an API for the inverse operator, it can be configured with a function to calculate the gradient, hereinafter referred to as the gradient calculation function. In this case, such as... Figure 7 As shown, the anomaly operator localization device 201 can define an API for inverse replacement by inheriting the gradient calculation function in the machine learning framework 200. The defined API can, for example, adopt a similar approach... Figure 6 The interface design of the API is shown (in this case, api_name in the API can be the name of the gradient calculation function). Furthermore, the anomaly operator localization device 201 can also apply the API of the forward operator in the machine learning framework 200 to the defined API. Further, when the machine learning framework 200 does not support applying the API of the forward operator to a custom API in the form of keyword arguments, the anomaly operator localization device 201 can pre-configure the passing order of each positional parameter in the custom API, and based on the passing order of each positional parameter, replace the keyword arguments corresponding to the API of the forward operator with the positional parameters at the specified positions in that passing order, thereby achieving the application of the API of the forward operator in the machine learning framework 200 to the defined API and realizing parameter passing.

[0110] For example, the API defined by the anomaly operator locator 201 may include, for example, such as Figure 8The diagram shows the forward and backward functions. When the target operator is reached, computation node 101 can call the defined API for backward replacement based on the accumulated gradient to request execution result 1. The accumulated gradient refers to the gradient passed to the target operator (i.e., the gradient calculated by the previous backward operator in the backpropagation sequence). Correspondingly, the persistence service on computation node 102 can receive the accumulated gradient passed when computation node 101 calls the API for backward replacement and run the forward function to obtain the input data of the forward operator required for backward replacement from computation node 101. Furthermore, the persistence service can also run the backward function to execute the forward operator using the input data, obtain the output of the forward operator, and calculate the corresponding gradient based on the output and accumulated gradient, thus obtaining the execution result 1 of the target operator. Finally, the persistence service can feed back the execution result 1 of the target operator to the computing node 101, so that the computing node 101 can use the execution result 1 to update the parameters in the forward operator and use the execution result 1 to calculate the gradient of the next backward operator, thereby completing the Nth round of training for the AI ​​model, without having to develop an additional backward operator in the machine learning framework 200.

[0111] For example, during the execution of the forward and reverse functions, such as Figure 9 As shown, the persistence service can run the forward function in this API, storing the input data of the forward operator through the context (CTX) manager. Then, the persistence service can run the inverse function in this API, retrieving the input data from the context manager and saving it to the video memory. Specifically, it can first save the data to the memory of compute node 102, and then save the input data from memory to the video memory. Furthermore, during the execution of the inverse function, the persistence service also obtains the accumulated gradient from compute node 101 and saves it to the video memory. Next, the persistence service can execute the forward operator corresponding to the target operator based on the input data in the video memory, obtaining the output result of the forward operator. Based on this output result and the accumulated gradient, it executes the gradient calculation function to calculate the gradient used to update the parameters, thus obtaining the execution result 1 of the target operator, as shown below. Figure 9 As shown. Since not all tensors in the input data need to have their gradients calculated, typically only tensors located in the leaf nodes of the computation graph need to have their gradients calculated. Therefore, computation node 101, while providing input data and accumulated gradients, can also provide gradient requirement information, such as... Figure 9As shown, this gradient requirement information could be, for example, a boolean array named "needs_input_grad" corresponding to the input data. This array indicates which tensors in the input data need gradients calculated and which do not. During the execution of the forward operator, the persistence service can add markers indicating whether gradients need to be calculated to the parameters in the forward operator. For example, it can set the value of `requires_grad` to "True" in the tensors of the input data in the forward operator. This value of `requires_grad` then instructs subsequent derivative calculations performed on the forward operator to obtain the corresponding gradients.

[0112] It is worth noting that the above explanation uses a single forward operator or a single backward operator as the target operator. In practical applications, the target operator may include multiple forward operators, multiple backward operators, or both forward and backward operators. Therefore, the anomaly operator localization device 201 can execute multiple target operators on the computing node 102 and complete the Nth round of training for the AI ​​model, as described in the various implementation examples above. Furthermore, when the target operator includes both a forward operator and its corresponding backward operator, the persistence service, during the execution of the backward operator, obtains the output of the forward operator by re-executing it, and further calculates the gradient based on this output. This improves the accuracy of gradient calculation for the backward operator compared to the persistence service using the output of the forward operator executed by the computing node 101 to calculate the gradient corresponding to the target operator.

[0113] In practical applications, the anomaly operator localization device 201 can instrument the AI ​​model to execute the target operator on the computing node 102 and use the obtained execution result 1 to participate in the Nth round of training of the AI ​​model.

[0114] In specific implementation, the anomaly operator locator 201 can, according to the target operator indicated in the configuration file, use a global replacement method to replace the code file of the target operator configured in the machine learning framework 200 with a new code file (or script). This new code file includes not only the program code of the target operator but also the program code that implements the use of the execution result 1 obtained from executing the target operator on the computing node 102 to participate in the Nth round of training of the AI ​​model. Thus, the anomaly operator locator 201 achieves forward / reverse replacement of the target operator by executing this new code file. For example, assuming the machine learning framework is specifically the PyTorch framework, the anomaly operator locator 201 can modify the __init__.py file in the PyTorch framework and add a new code file to the end of the file to replace the original code file of the target operator. Furthermore, in order not to affect the use of other code files, the anomaly operator locator 201 can set an environment variable in the machine learning framework 201, such as an environment variable named "TEMU_GLOBAL_FLAG". The anomaly operator locator 201 can configure the value of this environment variable to be true, so as to indicate that all target operator code files in the machine learning framework 200 are replaced with new code files.

[0115] Alternatively, the anomaly operator localization device 201 can modify the code file of the target operator configured in the machine learning framework 200 using a localized approach, based on the target operator indicated in the configuration file. Specifically, it can add an `enable()` function before the code segment of the target operator and a `disable()` function after the code segment of the target operator in the code file. The `enable()` and `disable()` functions are used to execute the target operator on the computing node 102 and use the execution result 1 to participate in the Nth round of training of the AI ​​model. In this way, the anomaly operator localization device 201 achieves forward / reverse replacement of the target operator by sequentially executing the program code in the code file.

[0116] S205: Anomaly operator localization device 201 compares the deviation between the loss value of the AI ​​model corresponding to execution result 1 and the reference loss value.

[0117] S206: When the deviation between the loss value of the AI ​​model corresponding to execution result 1 and the reference loss value meets the preset conditions, the anomaly operator localization device 201 determines that the target operator is an anomaly operator that affects the loss value during AI model training.

[0118] In this embodiment, after the execution result 1 obtained by executing the target operator using computing node 102 completes the Nth round of training of the AI ​​model, the anomaly operator location device 201 can observe the loss value generated during the AI ​​model training to determine whether the target operator affects the loss value of the AI ​​model. Specifically, when the target operator is a forward operator, the anomaly operator location device 201 can observe the loss value of the AI ​​model during the Nth round of training. When the target operator is a backward operator, the anomaly operator location device 201 can observe the loss value of the AI ​​model during the (N+1)th round of training. When the target operator includes both forward and backward operators, the anomaly operator location device 201 can observe the loss values ​​generated by the AI ​​model in both the Nth and (N+1)th rounds of training.

[0119] In specific implementation, the anomaly operator localization device 201 can compare the deviation between the loss value of the AI ​​model generated by execution result 1 and the reference loss value, and determine whether the target operator is an anomaly operator affecting the loss value of the AI ​​model based on the magnitude of the deviation. Furthermore, when it is determined that the target operator affects the loss value during AI model training, the anomaly operator localization device 201 can identify the target operator as an anomaly operator affecting the loss value during AI model training. Further, the anomaly operator localization device 201 can also feed back the corresponding localization result to the client, so that the client can notify the user that the target operator in the AI ​​model is an anomaly operator based on the localization result.

[0120] In this embodiment, the following two implementation examples are provided for determining whether a target operator is an abnormal operator.

[0121] In the first implementation example, before executing step S201, the AI ​​model can be pre-trained in both the training environment and the benchmark environment. By comparing the loss values ​​recorded in the logs of the training environment for each iteration with those recorded in the logs of the benchmark environment, anomaly operators affecting the AI ​​model's loss value are identified. When the target operator is a forward operator, the anomaly operator location device 201 can use the loss value of the Nth iteration recorded in the logs of the AI ​​model in the training environment as a reference loss value and calculate the difference between this reference loss value and the loss value corresponding to execution result 1. When the difference is within a preset range, it indicates that the target operator, when executed in the benchmark environment, does not cause a significant change in the AI ​​model's loss value. In this case, the anomaly operator location device 201 can determine that the target operator does not affect the AI ​​model's loss value. When the difference exceeds the preset range, it indicates that when the target operator is executed in the benchmark environment, it will cause a large change in the loss value of the AI ​​model. Thus, the abnormal operator location device 201 can determine that the deviation between the loss value corresponding to the execution result 1 and the reference loss value meets the preset condition, that is, determine that the target operator is an abnormal operator that affects the loss value of the AI ​​model.

[0122] Similarly, when the target operator is a reverse operator, the loss value corresponding to execution result 1 is the loss value generated by the AI ​​model in the (N+1)th round of training. Furthermore, the anomaly operator localization device 201 can use the loss value recorded in the AI ​​model's training environment logs in the (N+1)th round as a reference loss value. Then, the anomaly operator localization device 201 can calculate the difference between the reference loss value and the loss value corresponding to execution result 1. When the difference is within a preset range, the anomaly operator localization device 201 can determine that the target operator does not affect the AI ​​model's loss value; while when the difference exceeds the preset range, the anomaly operator localization device 201 can determine that the target operator is an anomaly operator that affects the AI ​​model's loss value.

[0123] In the second implementation example, before executing step S201, the AI ​​model can be pre-trained in both the training environment and the benchmark environment. By comparing the loss values ​​recorded in the logs of the training environment for each iteration of training with those recorded in the logs of the benchmark environment, an anomalous operator affecting the loss value of the AI ​​model is identified. When the target operator is a forward operator, the anomalous operator location device 201 can use the loss value of the Nth iteration recorded in the logs of the AI ​​model in the benchmark environment as a reference loss value and calculate the difference between this reference loss value and the loss value corresponding to execution result 1. When the difference is within a preset range, it indicates that when the target operator is executed in the benchmark environment, it will make the loss value of the AI ​​model similar to the overall loss value of the AI ​​model running in the benchmark environment, i.e., significantly different from the loss value of the AI ​​model running in the training environment. In this case, the anomalous operator location device 201 can determine that the deviation between the loss value corresponding to execution result 1 and the reference loss value meets a preset condition, i.e., the target operator is identified as an anomalous operator affecting the loss value of the AI ​​model. When the difference exceeds the preset range, it indicates that when the target operator is executed in the benchmark environment, the difference between the loss value of the AI ​​model and the loss value of the AI ​​model as a whole when running in the benchmark environment is still large. That is, the operator that causes a large difference in the loss value of the AI ​​model is not the target operator. At this time, the abnormal operator location device 201 can determine that the target operator does not affect the loss value of the AI ​​model.

[0124] Similarly, when the target operator is a reverse operator, the loss value corresponding to execution result 1 is the loss value generated by the AI ​​model in the (N+1)th round of training. Furthermore, the anomaly operator localization device 201 can use the loss value recorded in the AI ​​model's training environment logs in the (N+1)th round as a reference loss value. Then, the anomaly operator localization device 201 can calculate the difference between the reference loss value and the loss value corresponding to execution result 1. If the difference is within a preset range, the anomaly operator localization device 201 can determine that the target operator is an anomaly operator affecting the AI ​​model's loss value; and if the difference exceeds the preset range, the anomaly operator localization device 201 can determine that the target operator does not affect the AI ​​model's loss value.

[0125] Thus, by comparing the loss value corresponding to the execution result 1 with the reference loss value in the training environment / benchmark environment, the anomaly operator localization device 201 can verify whether the target operator is an anomaly operator, thereby effectively improving the accuracy of identifying anomaly operators.

[0126] In practical applications, the anomaly operator location device 201 can determine whether the deviation between the loss value corresponding to the execution result 1 and the reference loss value meets the preset conditions through other means. For example, the reference loss value can also be a loss value pre-configured by technicians (such as being manually set by technicians through theoretical derivation or based on technical experience).

[0127] It is worth noting that in this embodiment, the verification of whether the target operator is an anomalous operator is performed by executing the target operator on the computing node 102 during the Nth round of AI model training. In other possible embodiments, the anomalous operator localization device 201 can perform multiple rounds of iterative training on the AI ​​model, and execute the target operator on the computing node 102 in each round of training. In this way, the anomalous operator localization device 201 can comprehensively determine whether the target operator is an anomalous operator by observing the deviation between the loss value and the reference loss value during multiple rounds of training, thereby further improving the accuracy of identifying anomalous operators. The reference loss value used in different rounds of training can be different.

[0128] Furthermore, after identifying the target operators in the AI ​​model that affect the loss value, the anomaly operator localization device 201 can adjust the anomaly operators in the AI ​​model (or the execution result of the anomaly operators can be adjusted by the computing node 101) so that the loss value of the AI ​​model during iterative training in the training environment is approximately the same as the loss value of the AI ​​model during iterative training in the benchmark environment.

[0129] As an implementation example, the anomaly operator localization device 201 can obtain the execution results of anomaly operators in the training environment and the benchmark environment during multiple rounds of iterative training. For example, it can determine anomaly operators based on logs from both the training and benchmark environments. Then, based on the differences in the execution results of anomaly operators in different hardware environments, the anomaly operator localization device 201 adds compensation functions for the execution results of the anomaly operators. Thus, when the AI ​​model (including the anomaly operators) runs in the training environment, the anomaly operator localization device 201 can compensate for the execution results of the anomaly operators by executing the compensation functions. Figure 2 As shown, the results of the compensated anomaly operators are used in subsequent calculations. Thus, the results of the anomaly operators in the training environment, under the compensation function, remain similar to those in the benchmark environment, thereby ensuring that the loss value of the AI ​​model in the training environment is close to that in the benchmark environment.

[0130] It should be noted that the above Figure 2 In the illustrated embodiment, computing nodes 101 and 102, under the instruction of the anomaly operator localization device 201, complete the process of locating the anomaly operator in the AI ​​model. In other embodiments, computing nodes 101 and 102 may also actively execute the operation process of locating the anomaly operator in the AI ​​model without the need for instruction from other nodes or devices. The specific implementation process can be found in the description of the relevant parts of locating the anomaly operator in the above embodiments, and will not be repeated here.

[0131] The above Figure 2 In the illustrated embodiment, the process of executing the target operator on computing node 102 and feeding back the execution result 1 to computing node 101 can be completed by deploying a persistent service on computing node 102. The persistent service deployed on computing node 102 will be described below.

[0132] In one possible implementation, computing node 102 can utilize GPUs to execute the target operator, and multiple GPUs can be configured on computing node 102. Then, the anomaly operator location device 201 can utilize some of the GPUs on computing node 102 to start and run the persistence service. Thus, when computing node 101 calls the API provided by the persistence service, the persistence service can select a suitable GPU from the multiple GPUs to execute the target operator and obtain the corresponding execution result.

[0133] As a first implementation example, the persistence service can select a suitable GPU to execute the target operator based on the load of multiple GPUs. Specifically, when compute node 101 calls the API provided by the persistence service, the persistence service can obtain the load of multiple GPUs and determine the GPU with the lowest load by comparing the loads of these GPUs. The persistence service can then utilize this GPU to execute the target operator. For example, the GPU load can be represented by GPU utilization, and a higher GPU utilization indicates a higher GPU load. Thus, by utilizing the GPU with the lowest load to execute the target operator, the persistence service can balance the load of the multiple GPUs in compute node 102, improving the overall performance of compute node 102.

[0134] As a second implementation example, the persistence service can select GPUs to execute the target operator using a sequential allocation method. Specifically, multiple GPUs in compute node 102 can take turns executing the target operator in the AI ​​model. When compute node 101 calls the API provided by the persistence service, the persistence service can determine the GPU currently assigned to execute the task according to the allocation order of the multiple GPUs. For example, assuming compute node 102 includes 8 GPUs, and the 8 GPUs are labeled 0, 1, 2, 3, 4, 5, 6, and 7, if GPUs 0 through 5 have all been assigned a task (the task of executing the target operator) in the past time period, the persistence service can determine that GPU 6 is currently assigned to execute the target operator according to the allocation order (i.e., the order of increasing label size). Thus, the persistence service can utilize GPU 6 to execute the target operator and obtain the corresponding execution result.

[0135] In practical applications, when the training environment includes multiple computing nodes, these nodes can train the AI ​​model in parallel, such as training multiple modules within the same network layer of the AI ​​model. In this case, the multiple computing nodes in the training environment may call the persistent service on computing node 102 in parallel. To this end, the persistent service on computing node 102 can pre-create a process pool, which can include multiple processes. Furthermore, the persistent service can allocate the multiple processes in the process pool to different GPUs according to the GPU selection method described above, so that multiple GPUs can execute multiple processes in parallel, thereby achieving parallel execution of multiple target operators.

[0136] Furthermore, the training time for an AI model can be quite long, causing processes (such as those in a process pool) to continuously occupy GPU resources after being allocated by the persistence service. Therefore, for each GPU, the persistence service can also time the execution of the target operator. If, after a certain time (e.g., ten minutes), the persistence service still hasn't received a request to execute the target operator on that GPU, it indicates that the process's computing resources on the GPU have been idle for an extended period. In this case, the persistence service can shut down the process allocated to that GPU to release the computing (and storage) resources it has occupied. When the AI ​​model is undergoing the next training round, the persistence service will then reallocate the process to the GPU to execute the corresponding target operator. This prevents processes from remaining idle for extended periods and consuming GPU resources, thus avoiding impacting the overall performance of compute node 102. Furthermore, the hardware resource consumption in the benchmark environment is lightweight during the verification of anomaly operators in the AI ​​model. This allows the computing node 102 used to build the benchmark environment to perform other types of tasks, such as model training or model inference, based on other hardware resources, thereby improving the overall performance of the anomaly operator localization system 10.

[0137] It is worth noting that other reasonable combinations of steps that can be conceived by those skilled in the art based on the above description also fall within the scope of protection of this application. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to this application.

[0138] The above combination Figures 1a to 9 The anomaly operator localization method provided in the embodiments of this application will be introduced. Next, the structure of the anomaly operator localization device and computing device provided in the embodiments of this application will be described with reference to the accompanying drawings.

[0139] See Figure 10 The diagram shows a schematic of an anomaly operator localization device 1000, which includes:

[0140] The instruction module 1001 is used to instruct the first computing node to send the input data of the target operator in the AI ​​model to the second computing node during the Nth round of training of the AI ​​model, where N is a positive integer. The first computing node and the second computing node include different types of hardware. The hardware in the first computing node is used to run the AI ​​model, and the hardware in the second computing node is used to run the target operator. The module instructs the second computing node to execute the target operator according to the input data, obtain a first execution result, and send the first execution result to the first computing node. The module instructs the first computing node to continue the Nth round of training for the AI ​​model using the first execution result.

[0141] The determination module 1002 is used to determine the target operator as an abnormal operator that affects the loss value during AI model training when the deviation between the loss value of the AI ​​model corresponding to the first execution result and the reference loss value meets the preset conditions.

[0142] In one possible implementation, the anomaly operator location device 1000 further includes:

[0143] Module 1003 is used to obtain the configuration file, which includes the identifier of the target operator.

[0144] Deployment module 1004 is used to deploy the target operator on the second computing node according to the configuration file.

[0145] In one possible implementation, the target operator is a forward operator, and the loss value of the AI ​​model corresponding to the first execution result is the loss value generated by the AI ​​model during the Nth round of training.

[0146] The instruction module 1001 is specifically used to instruct the first computing node to package the input data of the forward operator into a file and send the file to the second computing node during the forward computation process of the Nth round of training of the AI ​​model.

[0147] In one possible implementation, the instruction module 1001 is specifically used for:

[0148] The first computing node is instructed to use the value in the first execution result to replace the value in the second execution result obtained by the first computing node in executing the target operator, so as to obtain the third execution result.

[0149] Instruct the first computing node to continue the Nth round of training for the AI ​​model based on the third execution result.

[0150] In one possible implementation, the target operator is a reverse operator, the first execution result is the gradient, and the loss value of the AI ​​model corresponding to the first execution result is the loss value generated by the AI ​​model during the N+1th round of training.

[0151] The instruction module 1001 is specifically used to: instruct the first computing node to send the accumulated gradient and the input data of the forward operator corresponding to the backward operator to the second computing node during the backward computation process of the Nth round of training of the AI ​​model. The accumulated gradient is the gradient calculated by the operator preceding the backward operator in the backpropagation sequence.

[0152] In one possible implementation, the instruction module 1001 is specifically used for:

[0153] The second computing node is instructed to execute the forward operator based on the input data of the forward operator and obtain the output result;

[0154] The second computation node is instructed to execute the inverse operator based on the output and the accumulated gradient to obtain the gradient.

[0155] In one possible implementation, the anomaly operator localization system further includes a machine learning framework, which includes a gradient computation function.

[0156] The instruction module 1001 is specifically used to instruct the second computing node to execute the gradient calculation function based on the output result and the accumulated gradient to obtain the gradient.

[0157] In one possible implementation, the machine learning framework includes the PyTorch framework or the TensorFlow framework.

[0158] because Figure 10 The anomaly operator location device 1000 shown corresponds to the above. Figure 2 The anomaly operator location device 201 in the illustrated embodiment, therefore Figure 10 For the specific implementation of the anomaly operator location device 1000 and its technical effects, please refer to the above. Figure 2 The relevant details in the illustrated embodiments are described in detail here, and will not be repeated here.

[0159] Figure 11 This application provides a schematic diagram of the hardware structure of a computing device 1100, which, for example, can implement the above-described... Figure 2 The abnormal operator location device 201 in the illustrated embodiment, etc.

[0160] like Figure 11As shown, the computing device 1100 includes a processor 1101, a memory 1102, and a communication interface 1103. The processor 1101, memory 1102, and communication interface 1103 communicate via a bus 1104, or via wireless transmission or other means. The memory 1102 stores instructions, and the processor 1101 executes the instructions stored in the memory 1102. Further, the computing device 1100 may also include a memory unit 1105, which is connected to the processor 1101, the storage medium 1102, and the communication interface 1103 via the bus 1104. The memory 1102 stores program code, and the processor 1101 can call the program code stored in the memory 1102 to perform the following operations:

[0161] The first computing node is instructed to send the input data of the target operator in the AI ​​model to the second computing node during the Nth round of training of the AI ​​model, where N is a positive integer. The first computing node and the second computing node include different types of hardware. The hardware in the first computing node is used to run the AI ​​model, and the hardware in the second computing node is used to run the target operator.

[0162] The second computing node is instructed to execute the target operator based on the input data to obtain a first execution result, and the first execution result is sent to the first computing node.

[0163] The first computing node is instructed to continue the Nth round of training for the AI ​​model using the first execution result.

[0164] When the deviation between the loss value of the AI ​​model corresponding to the first execution result and the reference loss value meets a preset condition, the target operator is determined to be an abnormal operator that affects the loss value during the training of the AI ​​model.

[0165] It should be understood that in this embodiment, the processor 1101 may be a CPU, but it may also be other general-purpose or special-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), graphics processing units (GPUs), neural network processors (NPUs), or other programmable logic devices, discrete gate or transistor logic devices, discrete device assemblies, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0166] The memory 1102 may include read-only memory and random access memory, and provides instructions and data to the processor 1101. The memory 1102 may also include non-volatile random access memory.

[0167] The memory 1102 can be volatile memory or non-volatile memory, or it can include both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0168] The communication interface 1103 is used to communicate with other devices connected to the computing device 1100. The bus 1104 may include a data bus, as well as a power bus, a control bus, and a status signal bus. However, for clarity, all buses are labeled as bus 1104 in the figure.

[0169] It should be understood that the computing device 1100 according to the embodiments of this application may correspond to the anomaly operator location device 1000 in the embodiments of this application, and may correspond to the execution of the embodiment of this application. Figure 2 The method executed by the anomaly operator locator 201 in the illustrated method, and the above-mentioned and other operations and / or functions implemented by the computing device 1100, are respectively for the purpose of implementing... Figure 2 The process of the corresponding methods in [the document] will not be elaborated here for the sake of brevity.

[0170] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute the above-described anomaly operator localization method.

[0171] This application also provides a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computing device, all or part of the processes or functions described in this application are generated.

[0172] The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, or data center to another website, computer, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0173] The computer program product can be a software installation package. When any of the aforementioned anomaly operator location methods is required, the computer program product can be downloaded and executed on a computing device.

[0174] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0175] The terminology used in the above embodiments is for the purpose of describing specific embodiments only and is not intended to be a limitation of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to also include expressions such as “one or more,” unless the context clearly indicates otherwise. It should also be understood that in the embodiments of this application, “one or more” refers to one, two, or more; the character “ / ” generally indicates that the preceding and following objects are in an “or” relationship. In the embodiments of this application, “simultaneously” means within the same time period, including situations where they are at the same moment. The terms “first,” “second,” etc., in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate, and this is merely a way of distinguishing objects with the same attributes in the embodiments of this application.

[0176] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0177] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An anomalous operator positioning method, characterized by, The method is applied to an abnormal operator positioning system, the abnormal operator positioning system comprising a first computing node and a second computing node, the first computing node and the second computing node comprising different types of hardware, the hardware in the first computing node being used to run an artificial intelligence (AI) model, and the hardware in the second computing node being used to run a target operator in the AI model. The method comprises: The first computing node sends input data of the target operator to the second computing node in an Nth round of training of the AI model, N being a positive integer. The second computing node executes the target operator according to the input data to obtain a first execution result, and sends the first execution result to the first computing node. The first computing node continues to execute the Nth round of training of the AI model by using the first execution result. When a deviation between a loss value of the AI model corresponding to the first execution result and a reference loss value satisfies a preset condition, the first computing node determines that the target operator is an abnormal operator affecting the loss value of the AI model in training.

2. The method of claim 1, wherein, The method further comprises: The first computing node obtains a configuration file, the configuration file comprising an identifier of the target operator. The first computing node deploys the target operator on the second computing node according to the configuration file.

3. The method according to claim 1 or 2, characterized in that, The target operator is a forward operator, and the loss value of the AI model corresponding to the first execution result is a loss value generated by the AI model in the Nth round of training. The first computing node sends input data of the target operator to the second computing node in an Nth round of training of the AI model, comprising: The first computing node packs the input data of the forward operator into a file in a forward calculation process of the Nth round of training of the AI model, and sends the file to the second computing node.

4. The method of claim 3, wherein, The first computing node continues to execute the Nth round of training of the AI model by using the first execution result, comprising: The first computing node replaces values in a second execution result obtained by the first computing node by executing the target operator with values in the first execution result to obtain a third execution result. The first computing node continues to execute the Nth round of training of the AI model based on the third execution result.

5. The method according to claim 1 or 2, characterized in that, The target operator is a backward operator, the first execution result is a gradient, and the loss value of the AI model corresponding to the first execution result is a loss value generated by the AI model in an (N+1)th round of training. The first computing node sends input data of the target operator to the second computing node in an Nth round of training of the AI model, comprising: The first computing node sends accumulated gradients and input data of a forward operator corresponding to the backward operator to the second computing node in a backward calculation process of the Nth round of training of the AI model, the accumulated gradients being gradients calculated by a previous operator of the backward operator in a backward propagation order.

6. The method of claim 5, wherein, The second computing node executes the target operator according to the input data to obtain a first execution result, including: The second computing node executes the forward operator according to input data of the forward operator to obtain an output result; The second computing node executes the backward operator according to the output result and the accumulated gradient to obtain the gradient.

7. The method of claim 6, wherein, The abnormal operator positioning system further comprises a machine learning framework, and the machine learning framework comprises a gradient calculation function; The second computing node executes the backward operator according to the output result and the accumulated gradient to obtain the gradient, including: The second computing node executes the gradient calculation function according to the output result and the accumulated gradient to obtain the gradient.

8. The method of claim 7, wherein, The machine learning framework comprises a PyTorch framework or a TensorFlow framework.

9. The method according to any one of claims 1 to 8, characterized in that, The method further comprises: When a duration that hardware for executing the target operator in the second computing node is idle reaches a preset duration, the second computing node releases a computing resource corresponding to the hardware for executing the target operator.

10. An anomalous operator positioning method, characterized by, The method comprises being executed by an abnormal operator positioning apparatus, and the method comprises: indicating a first computing node to send input data of a target operator in an artificial intelligence (AI) model to a second computing node in an Nth round of training of the AI model, where N is a positive integer, the first computing node and the second computing node comprise different types of hardware, hardware in the first computing node is used to run the AI model, and hardware in the second computing node is used to run the target operator; indicating the second computing node to execute the target operator according to the input data to obtain a first execution result, and to send the first execution result to the first computing node; indicating the first computing node to continue to execute the Nth round of training of the AI model by using the first execution result; when a deviation between a loss value of the AI model corresponding to the first execution result and a reference loss value satisfies a preset condition, determining that the target operator is an abnormal operator that affects a loss value of the AI model in training.

11. The method of claim 10, wherein, The method further comprises: obtaining a configuration file, where the configuration file comprises an identifier of the target operator; deploying the target operator on the second computing node according to the configuration file.

12. The method according to claim 10 or 11, characterized in that, The target operator is a forward operator, and the loss value of the AI model corresponding to the first execution result is a loss value generated by the AI model in the Nth round of training; The indication of the first computing node to send the input data of the target operator to the second computing node in the Nth round of training of the AI model comprises: indicating the first computing node to pack input data of the forward operator into a file in a forward calculation process of the Nth round of training of the AI model, and to send the file to the second computing node.

13. The method of claim 12, wherein, The indication of the first computing node to continue to execute the Nth round of training of the AI model by using the first execution result comprises: indicate the first computing node to replace values in second execution results obtained by the first computing node executing the target operator, with values in the first execution results, to obtain third execution results; indicate the first computing node to continue executing the Nth round of training process for the AI model based on the third execution results.

14. The method of claim 10 or 11, wherein, The target operator is a reverse operator, the first execution result is a gradient, and the loss value of the AI model corresponding to the first execution result is a loss value generated by the AI model in an N+1th round of training process. The indication of the first computing node to send input data of the target operator to the second computing node in the Nth round of training process of the AI model comprises: In the reverse calculation process of the Nth round of training of the AI model, the first computing node sends accumulated gradients and input data of a forward operator corresponding to the reverse operator to the second computing node, and the accumulated gradients are gradients calculated by a previous operator of the reverse operator in a reverse propagation sequence.

15. The method of claim 14, wherein, The indication of the second computing node to execute the target operator according to the input data to obtain first execution results comprises: The indication of the second computing node to execute the forward operator according to the input data of the forward operator to obtain output results; The indication of the second computing node to execute the reverse operator according to the output results and the accumulated gradients to obtain the gradients.

16. The method of claim 15, wherein, The abnormal operator positioning system further comprises a machine learning framework, and the machine learning framework comprises a gradient calculation function. The indication of the second computing node to execute the reverse operator according to the output results and the accumulated gradients to obtain the gradients comprises: The indication of the second computing node to execute the gradient calculation function according to the output results and the accumulated gradients to obtain the gradients.

17. The method of claim 16, wherein, The machine learning framework comprises a PyTorch framework or a TensorFlow framework.

18. An anomaly operator positioning apparatus characterized by comprising: The abnormal operator positioning apparatus comprises: An indication module is configured to indicate the first computing node to send input data of a target operator in an AI model to a second computing node in an Nth round of training process of the AI model, the N is a positive integer, the first computing node and the second computing node comprise different types of hardware, hardware in the first computing node is configured to run the AI model, and hardware in the second computing node is configured to run the target operator; indicate the second computing node to execute the target operator according to the input data to obtain first execution results, and send the first execution results to the first computing node; and indicate the first computing node to continue executing the Nth round of training process for the AI model based on the first execution results. A determination module is configured to determine that the target operator is an abnormal operator affecting a loss value of the AI model in training when a deviation between a loss value of the AI model corresponding to the first execution results and a reference loss value satisfies a preset condition.

19. An anomaly operator positioning system, characterized by, The abnormal operator positioning system comprises a first computing node and a second computing node, the first computing node and the second computing node comprise different types of hardware, the hardware in the first computing node is used to run an artificial intelligence (AI) model, and the hardware in the second computing node is used to run a target operator in the AI model; The first computing node is configured to send input data of the target operator to the second computing node in an Nth round of training of the AI model, where N is a positive integer; The second computing node is configured to execute the target operator according to the input data to obtain a first execution result, and send the first execution result to the first computing node; The first computing node is further configured to continue to execute the Nth round of training of the AI model by using the first execution result. When a deviation between a loss value of the AI model corresponding to the first execution result and a reference loss value satisfies a preset condition, the target operator is determined as an abnormal operator affecting the loss value of the AI model during training.

20. A computing device, comprising: The computing device comprises a processor and a memory; The memory is configured to store instructions, and the processor executes the instructions stored in the memory to enable the computing device to perform a method performed by the first computing node in any one of claims 1 to 9, or perform a method performed by the second computing node in any one of claims 1 to 9, or perform the method in any one of claims 10 to 17.

21. A computer-readable storage medium, characterized in that, The instructions, when executed on a computing device, enable the computing device to perform a method performed by the first computing node in any one of claims 1 to 9, or perform a method performed by the second computing node in any one of claims 1 to 9, or perform the method in any one of claims 10 to 17.

22. A computer program product comprising instructions, wherein: The instructions, when executed on at least one computing device, enable the at least one computing device to perform a method performed by the first computing node in any one of claims 1 to 9, or perform a method performed by the second computing node in any one of claims 1 to 9, or perform the method in any one of claims 10 to 17.

Images