Anomaly detection method, device and equipment based on pipeline parallel training strategy

Abstract

The invention provides an anomaly detection method, device and equipment based on a pipeline parallel training strategy, and relates to the field of artificial intelligence, in particular to the field of deep learning. According to the specific implementation scheme, in the process that an execution device trains a to-be-trained model, if it is detected that abnormal data exist in tensor parameters of the execution device in the training process, abnormal information is sent to a management and control device, and the execution device is controlled to stop executing the training process.

Description

Technical Field

[0001] The present disclosure relates to the field of deep learning in the field of artificial intelligence, and in particular to an anomaly detection method, apparatus and device based on a pipeline parallel training strategy. Background Art

[0002] With the development of artificial intelligence technology, more and more deep learning models are being used in the field of artificial intelligence. Before a deep learning model is put into use, it usually requires a lot of training using massive amounts of data. If an abnormality occurs during model training, it is usually necessary to perform anomaly detection on the execution device, model framework, etc. during the training process to identify the problem.

[0003] However, the anomaly detection process has the problem of low detection efficiency. Therefore, there is an urgent need for a solution that can effectively and timely perform anomaly detection in the model training process. Summary of the invention

[0004] The present disclosure provides a method, apparatus and device for anomaly detection based on a pipeline parallel training strategy.

[0005] According to a first aspect of the present disclosure, there is provided an anomaly detection method based on a pipeline parallel training strategy, comprising:

[0006] In a training process of training a to-be-trained model based on an execution device in a device set, if it is detected that a tensor parameter of the execution device in the training process has abnormal data, an abnormal information is sent to a control device; wherein the device set includes a plurality of execution devices; and the abnormal information indicates that a training abnormality occurs in the execution device;

[0007] Control the execution device to stop executing the training process.

[0008] According to a second aspect of the present disclosure, there is provided an anomaly detection device based on a pipeline parallel training strategy, comprising:

[0009] A detection unit, configured to send abnormal information to a control device during a training process of training a to-be-trained model based on an execution device in a device set, if it is detected that a tensor parameter of the execution device in the training process has abnormal data; wherein the device set includes a plurality of execution devices; and the abnormal information indicates that a training abnormality occurs in the execution device;

[0010] A control unit is used to control the execution device to stop executing the training process.

[0011] According to a third aspect of the present disclosure, there is provided an anomaly detection system based on a pipeline parallel training strategy, comprising a device set and a management and control device, wherein the device set comprises a plurality of execution devices; wherein the execution device is used to execute the method described in the first aspect.

[0012] According to a fourth aspect of the present disclosure, there is provided an electronic device, including:

[0013] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in the first aspect.

[0014] According to a fifth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method described in the first aspect.

[0015] According to a sixth aspect of the present disclosure, a computer program product is provided, comprising: a computer program, wherein the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and the at least one processor executes the computer program so that the electronic device executes the method described in the first aspect.

[0016] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0018] Figure 1 is a schematic diagram according to a first embodiment of the present disclosure;

[0019] Figure 2 is a schematic diagram of a scenario in which the first pipeline task of the embodiment of the present disclosure can be implemented;

[0020] Figure 3 is a schematic diagram of a scenario in which the second pipeline task of the embodiment of the present disclosure can be implemented;

[0021] Figure 4 is a schematic diagram according to a second embodiment of the present disclosure;

[0022] Figure 5 is a schematic diagram according to a third embodiment of the present disclosure;

[0023] Figure 6 is a schematic diagram according to a fourth embodiment of the present disclosure;

[0024] Figure 7 is a schematic diagram according to a fifth embodiment of the present disclosure;

[0025] Figure 8 is a schematic diagram according to a sixth embodiment of the present disclosure;

[0026] Fig. 9 is a schematic block diagram of an electronic device that can implement an embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0028] In the training process of deep learning, in order to further improve the training efficiency of the model, a distributed model training method is usually used. However, the current distributed training method usually arranges the model in multiple execution devices in the device set, and the management and control device divides the training data and distributes the divided training data to each execution device for training. Although this method improves the efficiency of model training through distributed devices, for the anomalies in the model training process, it can only reversely deduce after the anomaly occurs in the model training to determine the cause of the anomaly. For example, a stress test (stress test) is used to locate the faulty device that generates NaN (Not a Number) or Inf (Infinity). Obviously, this method for handling exceptions in the model training process not only has the problems of low anomaly detection efficiency and low anomaly handling efficiency, but also requires interrupting training and performing full equipment detection, which will seriously affect the training efficiency of the model and cause a significant decrease in training efficiency. In addition, the phenomenon of NaN or Inf may not be reproduced through stress testing, which increases the difficulty of locating faulty devices and further leads to a waste of computing resources and time. Therefore, there is an urgent need for a method that can detect and accurately locate abnormal faulty equipment online without affecting the training task.

[0029] Among them, the abnormalities in the model training process may be caused by hardware failure, memory error, data corruption, or model parameter abnormalities. For example, abnormal values ​​such as NaN or Inf may appear in the tensor parameters. These abnormal values ​​will cause the model to fail to update normally, affect the convergence of training, and even cause the training process to terminate.

[0030] To this end, the present application proposes a method for large-scale distributed model training using a pipeline parallel strategy. Based on the pipeline parallel strategy, the model can divide the layers of the model according to the number of execution devices for collaborative training to obtain multiple stages. According to the execution order of the layers of the model, the execution order of the multiple stages can be determined, and then the pipeline order of the multiple execution devices assigned to the multiple stages can be determined. The multiple execution devices can transmit the tensor parameters generated during the training process in the order of the pipeline. Each execution device can perform anomaly detection on the tensor parameters before transmitting the tensor parameters to the outside. If the tensor parameters are abnormal, the execution device can terminate the training in time and report the abnormality. Furthermore, the control device can replace the execution device with an idle execution device in the device set based on the abnormal execution device to facilitate the continued training of the model.

[0031] Specifically, the present disclosure is applicable to distributed training of ultra-large-scale pre-trained models such as BERT and GPT, computationally intensive model training in fields such as computer vision, natural language processing, and speech recognition, and training platforms that require long-term stable operation and high reliability requirements that are sensitive to faults.

[0032] Specifically, the present disclosure can perform anomaly detection immediately after executing the key computing steps of the device, so that anomalies can be discovered in time. It is also possible to check before sending tensor parameters to prevent abnormal data from propagating between devices and expanding the scope of influence. In addition, the present disclosure can accurately locate the computing device and specific computing node where the problem occurs by recording the device ID of the execution device and the tensor parameters where the anomaly occurs. In addition, based on the anomalies that occur, the present disclosure can interrupt abnormal calculations in time to avoid error accumulation and improve the stability and reliability of model training. In addition, adding a small amount of efficient detection at this key node has little effect on training performance, but greatly improves the effect of anomaly detection.

[0033] The present invention discloses an anomaly detection method, apparatus and device based on a pipeline parallel training strategy, which are applied to the deep learning field in the field of artificial intelligence to achieve the effect of effectively and timely detecting abnormal data in the model training process.

[0034] In order to enable readers to more deeply understand the implementation principle of the present disclosure, the following Figure 2-Figure 4 right Figure 1 The illustrated embodiment is further refined.

[0035] Figure 1 is a schematic diagram according to the first embodiment of the present disclosure, such as Figure 1 As shown, the present disclosure provides an anomaly detection method based on a pipeline parallel training strategy, the method comprising:

[0036] 101. During the training process of training a to-be-trained model based on an execution device in a device set, if it is detected that a tensor parameter of the execution device in the training process has abnormal data, an abnormal information is sent to a control device. The device set includes multiple execution devices. The abnormal information indicates that a training abnormality occurs in the execution device.

[0037] In this embodiment, the device set may include multiple execution devices. Some or all of the multiple execution devices are assigned a stage of the model. According to the execution order of each stage of the model, the data transmission order between the multiple execution devices can be determined, and this order is the pipeline order. When the multiple execution devices start model training, during the model training process, each execution device will process the input data according to the model parameters corresponding to the model stage stored in the execution device to obtain the tensor parameters to be output. The execution device can send the tensor parameters to other corresponding execution devices on the pipeline.

[0038] For example, when four execution devices are included, after the four execution devices are arranged in a pipeline order, the execution order of the training tasks in the four execution devices can be as follows: Figure 2 and Figure 3 As shown. Among them, Figure 2 It is a non-interleaved pipeline scheduling process. Figure 3 It is an interleaved pipeline scheduling process. Figure 2 and Figure 3 The pipeline scheduling process shown includes 8 micro-batches. Each micro-batch may include part of the training data. Each micro-batch needs to be processed in the forward sequence and the reverse sequence by 4 execution devices to realize the model training of the pipeline. In the sequential execution process of each micro-batch, after the previous execution device completes its calculation for the batch, it needs to send the calculation result to the next execution device so that the next execution device can perform subsequent calculations based on the calculation result of the previous execution device.

[0039] Before the execution device sends the tensor parameter to other execution devices, the execution device may detect the tensor parameter to determine whether the tensor parameter contains abnormal data. If the tensor parameter contains abnormal data, the execution device may generate abnormal information based on the abnormal data and send the abnormal information to the control device. The abnormal information is used to characterize the training abnormality of the execution device. Optionally, the abnormal information may include information such as the device ID of the execution device, the tensor name or identifier of the tensor parameter, the abnormal type, and a timestamp. Optionally, the execution device may include multiple tensor parameters to be detected. When one of the multiple tensor parameters contains abnormal data, the execution device generates abnormal information.

[0040] Optionally, if none of the multiple tensor parameters contain abnormal data, the execution device can output the tensor parameter according to other execution devices to which the tensor parameter needs to be output. Optionally, the multiple tensor parameters can be output to multiple other execution devices. Optionally, when there are multiple tensor parameters that need to be output to the same execution device, the multiple tensor parameters can be merged first. The execution device can output the merged multiple tensor parameters together to the corresponding execution device. The merged transmission of the parameters improves the efficiency of data transmission between execution devices and improves the training efficiency of the model.

[0041] Optionally, the abnormal data may be one or more of the following: data in a preset format, data with abnormal precision, and data with abnormal changes. The data in the preset format may be data such as NaN and Inf. The data with abnormal precision is data whose value is outside the precision range. The data with abnormal changes is data whose iteration difference does not meet the preset difference range.

[0042] Optionally, the tensor parameters may include one or more of the following: forward tensor parameters, reverse tensor parameters, parameter gradients, and model parameters. The forward tensor parameters are the calculation results of the forward propagation calculated by the execution device, and the executed device will output them forward to the next execution device according to the pipeline direction. The reverse tensor parameters are the calculation results of the reverse propagation calculated by the execution device, and the executed device will output them backward to the previous execution device according to the pipeline direction. The gradient parameters are used to indicate the gradient of changes in the model parameter training process. Model parameters are the trained parameters in the model to be trained.

[0043] 102. Control the execution device to stop executing the training process.

[0044] In this embodiment, when the execution device detects abnormal data, the execution device can be controlled to stop the execution of the current training process. Optionally, the execution device can stop sending the current tensor parameters.

[0045] In this embodiment, by detecting whether there is abnormal data in the tensor parameters output by the execution device, the execution device can promptly alarm, locate, and stop model training when abnormal data occurs, thereby avoiding further expansion of the abnormal data and causing abnormalities in the model parameters, and can promptly locate the execution device where the abnormality occurs, thereby improving the subsequent exception handling efficiency.

[0046] Figure 4 is a schematic diagram according to the second embodiment of the present disclosure, such as Figure 4As shown, the present disclosure provides an anomaly detection method based on a pipeline parallel training strategy. The method is used to detect abnormal data in tensor parameters. The abnormal data may include data in a preset format, data with abnormal accuracy, data with abnormal changes, etc. Based on these anomalies, the specific implementation of anomaly detection may include the following steps:

[0047] 401. If it is detected that the data of the tensor parameter has data in a preset format, it is determined that abnormal data is detected.

[0048] In this embodiment, a tensor detection function may be preset in the execution device. The tensor detection function may be used to detect data in a preset format in the tensor parameter. Optionally, the preset format may be NaN, Inf, etc. Optionally, the tensor detection function may be functions such as isnan() and isinf(). Optionally, the tensor detection function may be a function in an existing deep learning framework, or the tensor detection function may be a function customized as needed. The detection of abnormal data in the preset format may prevent the data in the abnormal format from being propagated during the model training process, resulting in abnormal model parameters.

[0049] 402. If it is detected that the data of the tensor parameter contains data that exceeds the precision abnormality, it is determined that abnormal data is detected.

[0050] In this embodiment, a tensor detection function may be preset in the execution device. The tensor detection function may also be used to detect whether the data in the tensor parameter has an accuracy anomaly. If there is data with an accuracy anomaly, it is determined that the abnormal data is detected.

[0051] Optionally, the process of detecting data with abnormal accuracy may include the following two steps:

[0052] 4021. Determine the accuracy range according to the current convergence degree information of the model to be trained.

[0053] 4022. If it is detected that there is data in the tensor parameter that is not within the precision range, then determine the data with precision abnormality in the data of the tensor parameter.

[0054] In this embodiment, the execution device may be preset with an accuracy range for accuracy anomaly detection. The tensor detection function may be used to match whether the data in the tensor parameter is within the accuracy range. If the data is within the accuracy range, it is determined that the accuracy of the value meets the requirements. If the value is not within the accuracy range, it is determined that the accuracy of the value does not meet the requirements and there is an accuracy anomaly. If there is data with an accuracy anomaly, it is determined that abnormal data is detected.

[0055] Optionally, the accuracy range can be determined based on the current convergence degree information of the model to be trained. For example, an initial numerical range can be preset in the execution device. When the model just starts to be trained, the initial numerical range can be used as the accuracy range. As the model is trained, the convergence degree of the model changes. According to the convergence degree of the model in the current iteration, the initial data range can be narrowed to obtain a smaller numerical range, and the smaller numerical range is used as the accuracy range. Optionally, the narrowing method can be to determine a reduction ratio based on the convergence degree. Determine the new numerical range based on the product of the reduction ratio and the initial data range. The dynamic adjustment of the accuracy range can further ensure the validity of the data during the model training process, avoid numerical jumps or the appearance of singular values, and cause the model to have abnormal accuracy in subsequent training.

[0056] Optionally, if the model to be trained is a model applied to speech recognition, the tensor parameter is a speech feature parameter in the model applied to speech recognition. The accuracy range can also be adjusted based on a preset frequency range of human voice, so as to further improve the validity of the data and process the model in a timely manner when invalid data or abnormal data is found.

[0057] 403. If the model to be trained is a model applied to image recognition, and the tensor parameter is an image feature parameter in the model applied to image recognition, the gradient information between pixels in the tensor parameter is obtained. If an abnormal change in the gradient information is detected, it is determined that abnormal data is detected.

[0058] In this embodiment, if the model to be trained is a model applied to image recognition, the tensor parameter is an image feature parameter in the model applied to image recognition. Based on the image feature parameter, the gradient information between the pixels can be obtained, and the gradient is detected to determine whether there is an abnormal change in the gradient information of each pixel. If there is an abnormal change, it is determined that abnormal data is detected.

[0059] Optionally, the abnormal change may specifically be whether the direction indicated by the gradient information is consistent. If consistent, it means that the pixel point is normal. If inconsistent, it means that the pixel point has an abnormal change, and the pixel point can be used as abnormal data. This detection method can be based on the model of image recognition to further improve the efficiency of model processing and avoid excessive processing of the image during the model training process, resulting in poor model training effect.

[0060] 404. Detect the iterative difference of the tensor parameters of the training model during the training process. The iterative difference represents the data difference between the last iteration and the current iteration of the tensor parameters during the training process; if it is determined whether the tensor parameters have abnormal changes, it is determined that abnormal data is detected.

[0061] In this embodiment, the execution device may also store tensor parameters in the previous iteration process. After calculating the tensor parameters in this iteration process, the difference between the tensor parameters of the local iteration and the tensor parameters of the previous iteration may be calculated. Based on the difference in the tensor parameters, it may be determined whether the tensor parameters have abnormal changes. If there are abnormal changes, it is determined that abnormal data has been detected.

[0062] Optionally, the detection of whether there is an abnormal change can be achieved by presetting a difference range. If the difference is within the preset difference range, the tensor parameter does not change abnormally during the iteration. If the difference is not within the preset range, the tensor parameter changes abnormally during the iteration, and it is determined that data abnormality is detected. The abnormality detection process can further ensure the stability of the tensor parameters during the iteration process, thereby improving the effectiveness of model training.

[0063] In this embodiment, multiple anomaly detection methods are used to detect multiple anomalies in the tensor parameters, thereby improving the accuracy of anomaly detection of the tensor parameters and ensuring the model training efficiency and model training effect.

[0064] On the basis of the above embodiment, the execution device can also send the model parameters of the model training team generated by the iteration to the management and control device for storage after completing one iteration of training. Optionally, the execution device can send the model parameters after each iteration. Or the execution device can send periodically according to a fixed cycle. For example, the execution device can send the model parameters once every five iterations. The sending of the model parameters can synchronize the model parameters to the replaced execution device when an abnormality occurs in the execution device, so that the replaced execution device can take over the abnormal execution device for subsequent training.

[0065] Figure 5 is a schematic diagram according to the third embodiment of the present disclosure, Figure 5 As shown, the present disclosure provides an anomaly detection method based on a pipeline parallel training strategy. In the process based on pipeline parallel training, the execution steps of the execution device and the anomaly detection steps in the execution process can be as follows, including:

[0066] 501. Forward calculation.

[0067] In this embodiment, on each execution device, the execution device may perform forward propagation calculation on the input data to obtain a forward tensor parameter. The forward tensor parameter may specifically be a forward activation value.

[0068] 502. Check the forward tensor parameters.

[0069] In this embodiment, by performing an abnormality check on the forward tensor parameter, it is determined whether there is abnormal data in the forward tensor, thereby ensuring the validity of the forward calculation result and preventing the generation and propagation of abnormal data. Optionally, the abnormality check at least includes a check on NaN or Inf values.

[0070] Optionally, the check of abnormal values ​​can be implemented by writing an efficient tensor check function. The tensor check function can quickly detect whether there is abnormal data in the tensor parameter. For example, abnormality detection for NaN or Inf values ​​can be implemented by isnan() and isinf() functions. Optionally, the function can be a custom function, or the function can be a function in the PyTorch or TensorFlow deep learning framework. Optionally, the abnormality detection can be implemented in Paddlepaddle's distributed model training environment.

[0071] Optionally, when abnormal data is detected, the execution device can immediately interrupt the current calculation and generate exception information. The exception information may at least include an identifier such as a device ID for identifying the execution device where the exception occurred, a timestamp of the time when the exception occurred, the name or identifier of the tensor where the exception occurred, and the exception type of NaN or Inf.

[0072] Optionally, the execution device may report the abnormal information to a control device. Optionally, the control device may be a central monitoring or log system. The control device may perform fault diagnosis based on the abnormal information.

[0073] 503. Send forward tensor parameters.

[0074] In this embodiment, the execution device can send the forward calculation tensor that has been checked and confirmed to be normal to the next device to continue the pipeline calculation. Optionally, the next device can be the next execution device of the execution device on the pipeline.

[0075] 504. Reverse calculation.

[0076] In this embodiment, the execution device may calculate the back propagation of the error to obtain a reverse tensor parameter. Optionally, the reverse tensor parameter includes gradient information.

[0077] 505. Check the reverse tensor parameters.

[0078] In this embodiment, in order to ensure the validity of the reverse tensor parameters calculated in reverse and prevent abnormal gradients in the reverse tensor parameters from causing parameter update errors, the reverse tensor parameters can be detected for abnormal data. The abnormal data detection process and the processing method when abnormal data is found are the same as the calculation process of the forward tensor parameters described above.

[0079] 506. Send reverse tensor parameters.

[0080] In this embodiment, the execution device sends the checked reverse tensor parameters to the next device. Optionally, the next device may be the previous execution device of the execution device on the pipeline.

[0081] 507. Circular judgment.

[0082] In this embodiment, if the forward and reverse calculations of each execution device on the pipeline have been completed, the subsequent steps are continued to be executed. Otherwise, a waiting state is entered until the forward and reverse calculations of each execution device are completed.

[0083] 508. Check parameter gradients.

[0084] In this embodiment, the execution device can ensure the validity of all gradient values ​​before updating the model parameters. Optionally, the execution device can detect abnormal data on the local training parameter gradient. The detection process and the processing method when abnormal data occurs are the same as the forward tensor parameters.

[0085] 509. Cross-device gradient communication.

[0086] In this embodiment, each execution device may communicate parameter gradients to achieve synchronization of the parameter gradients on each execution device, in preparation for updating the model parameters.

[0087] 510. Optimizer update.

[0088] In this embodiment, the execution device can use the synchronized parameter gradients to update the model parameters according to the optimizer algorithm.

[0089] In this embodiment, through the above process, the execution of the execution device in the parallel training process of the pipeline is realized, and the abnormal data of the tensor parameters is detected during the training process, so that when an abnormality occurs during the model training process, the abnormality can be located and processed in time to avoid the impact of the abnormality on the model training.

[0090] Figure 6 is a schematic diagram according to a fourth embodiment of the present disclosure, Figure 6 As shown, the present disclosure provides an anomaly detection device 600 based on a pipeline parallel training strategy, the device comprising:

[0091] The detection unit 610 is used for sending abnormal information to the control device during the training process of training the to-be-trained model based on the execution device in the device set, if it is detected that the tensor parameters of the execution device in the training process have abnormal data. The device set includes multiple execution devices. The abnormal information indicates that the execution device has a training abnormality.

[0092] The control unit 620 is used to control the execution device to stop executing the training process.

[0093] The device of this embodiment can execute the technical solution in the above method. Its specific implementation process and technical principles are the same and will not be repeated here.

[0094] Figure 7 is a schematic diagram according to a fifth embodiment of the present disclosure, Figure 7 As shown, the present disclosure provides an anomaly detection device 700 based on a pipeline parallel training strategy, the device comprising:

[0095] The detection unit 710 is used for sending abnormal information to the control device during the training process of training the to-be-trained model based on the execution device in the device set, if it is detected that the tensor parameters of the execution device in the training process have abnormal data. The device set includes multiple execution devices. The abnormal information indicates that the execution device has a training abnormality.

[0096] The control unit 720 is used to control the execution device to stop executing the training process.

[0097] Optionally, the abnormal data is one or more of the following: data in a preset format, data with abnormal accuracy, and data with abnormal changes.

[0098] Optionally, the detection unit 710 includes:

[0099] The first detection module 711 is used to determine that abnormal data is detected if data having a preset format is detected in the data of the tensor parameter based on a preset tensor detection function.

[0100] Optionally, the detection unit 710 includes:

[0101] The second detection module 712 is used to determine that abnormal data is detected if it is detected that the data of the tensor parameter has data exceeding the precision range based on the preset tensor detection function.

[0102] Optionally, the second detection module 712 includes:

[0103] The first parameter submodule 7121 is used to determine the accuracy range according to the current convergence degree information of the model to be trained.

[0104] Optionally, if the model to be trained is a model applied to speech recognition, the tensor parameters are speech feature parameters in the model applied to speech recognition. The second detection module 712 includes:

[0105] The second parameter submodule 7122 is used to adjust the accuracy range according to the preset frequency range of human voice.

[0106] Optionally, if the model to be trained is a model applied to image recognition, the tensor parameters are image feature parameters in the model applied to image recognition. The detection unit 710 includes:

[0107] The third detection module 713 is used to obtain the gradient information between the pixels in the tensor parameter. If it is detected that the directions indicated by the gradient information are inconsistent, it is determined that abnormal data is detected.

[0108] Optionally, the detection unit 710 includes:

[0109] The fourth detection module 714 is used to detect the iterative difference of the tensor parameters of the training model during the training process. The iterative difference represents the data difference between the last iteration and the current iteration of the tensor parameters during the training process. If it is determined that the iterative difference of the tensor parameters does not meet the preset difference range, it is determined that abnormal data is detected.

[0110] Optionally, the tensor parameters include one or more of the following: forward tensor parameters, reverse tensor parameters, parameter gradients, and model parameters. Among them, the forward tensor parameters are the calculation results of the forward propagation calculated by the execution device, and the executed device will output them forward to the next execution device according to the pipeline direction. The reverse tensor parameters are the calculation results of the reverse propagation calculated by the execution device, and the executed device will output them backward to the previous execution device according to the pipeline direction. The gradient parameters are used to indicate the gradient of changes in the model parameter training process. Model parameters are the trained parameters in the model to be trained.

[0111] Optionally, the abnormality detection device 700 further includes:

[0112] The sending unit 730 is used to send the model parameters updated during the training process of the model to be trained to the management and control device for storage.

[0113] Optionally, the abnormality detection device 700 further includes:

[0114] The merging unit 740 is used to merge multiple tensor parameters output to the same execution device and transmit the merged multiple tensor parameters if no abnormal data is detected in the tensor parameters of the execution device during the training process of training the to-be-trained model based on the execution device in the device set.

[0115] The device of this embodiment can execute the technical solution in the above method. Its specific implementation process and technical principles are the same and will not be repeated here.

[0116] Figure 8 is a schematic diagram according to a sixth embodiment of the present disclosure, Figure 8As shown, the present disclosure provides an anomaly detection system 800 based on a pipeline parallel training strategy, the system comprising:

[0117] The device set 810 and the control device 820 include a plurality of execution devices 811. The execution device is used to execute the above Figures 1 to 3 The anomaly detection method based on the pipeline parallel training strategy in the illustrated embodiment. The control device is used to determine the replacement execution device from the execution devices in the device set that are in an idle state based on the anomaly information. And the model parameters of the execution device with the anomaly information are sent to the alternative execution device, so that the replacement execution device takes over the execution device with the anomaly information to participate in the training process based on the model parameters.

[0118] In this embodiment, the management and control device schedules the execution device with abnormal information and the replacement execution device, thereby further ensuring the training efficiency of the model, so that the model training will not be affected by the abnormal data, and improving the efficiency of abnormal processing.

[0119] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0120] According to an embodiment of the present disclosure, the present disclosure also provides a computer program product, which includes: a computer program, the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device executes the solution provided by any of the above embodiments.

[0121] Fig. 9 A schematic block diagram of an example electronic device 900 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0122] like Fig. 9As shown, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0123] A number of components in the device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0124] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, such as an anomaly detection method based on a pipeline parallel training strategy. For example, in some embodiments, an anomaly detection method based on a pipeline parallel training strategy may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the anomaly detection method based on the pipeline parallel training strategy described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to execute the anomaly detection method based on the pipeline parallel training strategy in any other appropriate manner (eg, by means of firmware).

[0125] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0126] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0127] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0128] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0129] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0130] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0131] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0132] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. An anomaly detection method based on pipeline parallel training strategy, comprising: In a training process of training a to-be-trained model based on an execution device in a device set, if it is detected that a tensor parameter of the execution device in the training process has abnormal data, an abnormal information is sent to a control device; wherein the device set includes a plurality of execution devices; and the abnormal information indicates that a training abnormality occurs in the execution device; Control the execution device to stop executing the training process.

2. The method according to claim 1, wherein the abnormal data comprises data in a preset format; detecting that the tensor parameters of the execution device during training have abnormal data comprises: If it is detected that the data of the tensor parameter has data in a preset format, it is determined that the abnormal data is detected.

3. The method according to claim 1, wherein the abnormal data comprises data with abnormal accuracy; detecting that the tensor parameters of the execution device during training have abnormal data comprises: If data with abnormal precision is detected in the data of the tensor parameter, it is determined that the abnormal data is detected.

4. The method according to claim 3, wherein the detecting of data with abnormal accuracy in the data of the tensor parameter comprises: Determining the accuracy range according to the current convergence degree information of the model to be trained; If it is detected that there is data in the tensor parameter that is not within the precision range, data with abnormal precision in the data of the tensor parameter is determined.

5. According to the method of claim 4, if the model to be trained is a model applied to speech recognition, the tensor parameter is a speech feature parameter in the model applied to speech recognition; the method further comprises: The accuracy range is adjusted according to a preset frequency range of human voice.

6. The method according to claim 1, wherein the abnormal data comprises data with abnormal changes; if the model to be trained is a model applied to image recognition, the tensor parameter is an image feature parameter in the model applied to image recognition; Detecting that the execution device has abnormal data in tensor parameters during training, including: Obtaining gradient information between pixels in the tensor parameter; If it is detected that the gradient information has an abnormal change, it is determined that the abnormal data is detected.

7. The method according to claim 1, wherein the abnormal data comprises data with abnormal changes; Detecting that the execution device has abnormal data in tensor parameters during training, including: Detecting the iterative difference of the tensor parameter of the training model during the training process; wherein the iterative difference represents the data difference between the last iteration and the current iteration of the tensor parameter during the training process; If it is determined that there is an abnormal change in the iterative difference of the tensor parameter, it is determined that the abnormal data is detected.

8. The method according to any one of claims 1 to 7, wherein the tensor parameters include one or more of the following: forward tensor parameters, reverse tensor parameters, parameter gradients, model parameters; in, The forward tensor parameter is a calculation result of the forward propagation calculated by the execution device, and will be output forward to the next execution device by the execution device according to the pipeline direction; The reverse tensor parameter is a calculation result of reverse propagation calculated by the execution device, and will be outputted by the execution device in reverse to the previous execution device according to the pipeline direction; The gradient parameter is used to indicate the change gradient of the model parameter during the training process; the model parameter is the trained parameter in the model to be trained.

9. The method according to any one of claims 1 to 8, further comprising: During the training process of the model to be trained, the model parameters updated during the training process are sent to the control device for storage.

10. The method according to any one of claims 1 to 9, further comprising: During the training process of training a model to be trained based on an execution device in a device set, if no abnormal data is detected in the tensor parameters of the execution device during the training process, multiple tensor parameters output to the same execution device are merged and the merged multiple tensor parameters are transmitted.

11. An anomaly detection device based on a pipeline parallel training strategy, comprising: A detection unit, configured to send abnormal information to a control device during a training process of training a to-be-trained model based on an execution device in a device set, if it is detected that a tensor parameter of the execution device in the training process has abnormal data; wherein the device set includes a plurality of execution devices; and the abnormal information indicates that a training abnormality occurs in the execution device; A control unit is used to control the execution device to stop executing the training process.

12. An anomaly detection system based on a pipeline parallel training strategy, the system comprising a device set and a control device, the device set comprising a plurality of execution devices; wherein, The execution device is used to execute the method according to any one of claims 1 to 10.

13. According to the system of claim 12, the management and control device is used to determine a replacement execution device from the execution devices in the device set that are in an idle state based on the exception information; and send model parameters of the execution device with the exception information to the alternative execution device, so that the replacement execution device takes over the execution device with the exception information to participate in the training process based on the model parameters.

14. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 10.

15. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-10.

16. A computer program product, comprising a computer program, which implements the steps of the method according to any one of claims 1 to 10 when executed by a processor.

Images