Anomaly detection method, device and equipment based on pipeline parallel training strategy

Abstract

This disclosure provides an anomaly detection method, apparatus, and device based on a pipeline parallel training strategy, relating to the field of artificial intelligence, particularly deep learning. A specific implementation scheme involves: during the training of a model by an execution device, if abnormal data is detected in a tensor parameter of the execution device during training, the execution device sends an abnormality message to a control device, and controls the execution device to stop the training process.

Description

Technical Field

[0001] The present disclosure relates to the field of deep learning in the field of artificial intelligence, and in particular to an anomaly detection method, apparatus, and device based on a pipeline parallel training strategy. Background Art

[0002] With the development of artificial intelligence (AI) technology, more and more deep learning models are being applied in AI applications. Before they can be put into use, deep learning models typically require extensive training using massive amounts of data. If anomalies occur during model training, it is often necessary to perform anomaly detection on the execution devices and model framework during the training process to identify the underlying issue.

[0003] However, the anomaly detection process has the problem of low detection efficiency. Therefore, there is an urgent need for a solution that can effectively and timely perform anomaly detection during the model training process. Summary of the Invention

[0004] The present disclosure provides a method, apparatus, and device for anomaly detection based on a pipeline parallel training strategy.

[0005] According to a first aspect of the present disclosure, a method for anomaly detection based on a pipeline parallel training strategy is provided, comprising:

[0006] During a training process of a to-be-trained model being trained by an execution device in a device set, if abnormal data is detected in a tensor parameter of the execution device during the training process, an abnormality message is sent to a control device; wherein the device set includes a plurality of execution devices; and the abnormality message indicates that a training abnormality has occurred in the execution device;

[0007] The execution device is controlled to stop executing the training process.

[0008] According to a second aspect of the present disclosure, there is provided an anomaly detection device based on a pipeline parallel training strategy, comprising:

[0009] a detection unit configured to, during a training process of a to-be-trained model being trained by an execution device in a device set, send an abnormality message to a control device if abnormal data is detected in a tensor parameter of the execution device during training; wherein the device set includes a plurality of execution devices; and the abnormality message indicates that a training abnormality has occurred in the execution device;

[0010] A control unit is used to control the execution device to stop executing the training process.

[0011] According to a third aspect of the present disclosure, an anomaly detection system based on a pipeline parallel training strategy is provided, comprising a device set and a management and control device, wherein the device set comprises a plurality of execution devices; wherein the execution device is used to execute the method described in the first aspect.

[0012] According to a fourth aspect of the present disclosure, there is provided an electronic device, including:

[0013] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method described in the first aspect.

[0014] According to a fifth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method described in the first aspect.

[0015] According to a sixth aspect of the present disclosure, a computer program product is provided, comprising: a computer program, wherein the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and the at least one processor executes the computer program so that the electronic device executes the method described in the first aspect.

[0016] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0018] Figure 1 is a schematic diagram according to a first embodiment of the present disclosure;

[0019] Figure 2 is a schematic diagram of a scenario in which the first pipeline task of an embodiment of the present disclosure can be implemented;

[0020] Figure 3 is a schematic diagram of a scenario in which the second pipeline task of the embodiment of the present disclosure can be implemented;

[0021] Figure 4 is a schematic diagram according to a second embodiment of the present disclosure;

[0022] Figure 5 is a schematic diagram according to a third embodiment of the present disclosure;

[0023] Figure 6 is a schematic diagram according to a fourth embodiment of the present disclosure;

[0024] Figure 7 is a schematic diagram according to a fifth embodiment of the present disclosure;

[0025] Figure 8 is a schematic diagram according to a sixth embodiment of the present disclosure;

[0026] Figure 9 is a schematic block diagram of an electronic device that can implement an embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0028] During deep learning training, distributed model training methods are often used to further improve model training efficiency. However, current distributed training methods typically deploy the model across multiple execution devices in a device collection. The management and control device then partitions the training data and distributes the partitioned training data to each execution device for training. Although this approach improves model training efficiency through distributed devices, for exceptions during model training, the only way to determine the cause of the exception is to reversely deduce after the exception occurs during model training. For example, stress testing (stress testing) is used to locate faulty devices that generate NaN (Not a Number) or Inf (Infinity). Obviously, this approach to handling exceptions during model training not only has the problems of low exception detection and handling efficiency, but also requires interrupting training and performing full device testing, which seriously affects the model training efficiency and significantly reduces training efficiency. In addition, stress testing may not necessarily reproduce the NaN or Inf phenomenon, which increases the difficulty of locating the faulty device and further wastes computing resources and time. Therefore, there is an urgent need for a method that can detect and accurately locate abnormal faulty equipment online without affecting the training task.

[0029] Exceptions during model training may be caused by hardware failure, memory errors, data corruption, or model parameter anomalies. For example, abnormal values ​​such as NaN or Inf may appear in tensor parameters. These abnormal values ​​can prevent the model from updating properly, affecting training convergence and even causing training to terminate.

[0030] To this end, the present application proposes a method for large-scale distributed model training using a pipeline parallel strategy. Based on the pipeline parallel strategy, the model can divide the layers of the model according to the number of execution devices for collaborative training to obtain multiple stages. According to the execution order of the layers of the model, the execution order of the multiple stages can be determined, and then the pipeline order of the multiple execution devices assigned to the multiple stages can be determined. The multiple execution devices can transmit the tensor parameters generated during the training process in the order of the pipeline. Each execution device can perform an anomaly detection on the tensor parameter before transmitting the tensor parameter to the outside. If there is an anomaly in the tensor parameter, the execution device can terminate the training in time and report the anomaly. Furthermore, the control device can replace the execution device with an idle execution device in the device set based on the execution device where the anomaly occurs, so as to facilitate continued training of the model.

[0031] Specifically, the present disclosure is applicable to the distributed training of ultra-large-scale pre-trained models such as BERT and GPT, computationally intensive model training in fields such as computer vision, natural language processing, and speech recognition, and training platforms with high reliability requirements that require long-term stable operation and are sensitive to faults.

[0032] Specifically, the present disclosure can perform anomaly detection immediately after executing the key computing steps of the device, so that anomalies can be discovered in time. It can also check before sending tensor parameters to prevent abnormal data from spreading between devices and expanding the scope of influence. In addition, the present disclosure can accurately locate the computing device and specific computing node where the problem occurs by recording the device ID of the executing device and the tensor parameter where the anomaly occurs. In addition, based on the anomaly that occurs, the present disclosure can interrupt the abnormal calculation in time, avoid error accumulation, and improve the stability and reliability of model training. In addition, adding a small amount of efficient detection at this key node has little effect on training performance, but greatly improves the effect of anomaly detection.

[0033] The present disclosure provides an anomaly detection method, apparatus, and device based on a pipeline parallel training strategy, which are applied to the field of deep learning in the field of artificial intelligence to achieve the effect of effectively and timely detecting abnormal data during the model training process.

[0034] In order to enable readers to have a deeper understanding of the implementation principle of this disclosure, the following Figure 2-Figure 4 right Figure 1 The illustrated embodiment is further refined.

[0035] Figure 1 is a schematic diagram according to the first embodiment of the present disclosure, as shown in Figure 1 As shown, the present disclosure provides an anomaly detection method based on a pipeline parallel training strategy, the method comprising:

[0036] 101. During a training process of a to-be-trained model using an execution device in a device set, if abnormal data is detected in a tensor parameter of the execution device during training, an abnormality message is sent to a control device. The device set includes multiple execution devices. The abnormality message indicates that a training abnormality occurred on the execution device.

[0037] In this embodiment, the device set may include multiple execution devices. Some or all of the multiple execution devices are assigned to a stage of the model. According to the execution order of each stage of the model, the data transmission order between the multiple execution devices can be determined, and this order is the pipeline order. After the multiple execution devices start model training, during the model training process, each execution device will process the input data according to the model parameters corresponding to the model stage stored in the execution device to obtain the tensor parameters to be output. The execution device can send the tensor parameters to other corresponding execution devices on the pipeline.

[0038] For example, when there are four execution devices, after the four execution devices are arranged in a pipeline order, the execution order of the training tasks in the four execution devices can be as follows: Figure 2 and Figure 3 As shown. Among them, Figure 2 It is a non-interleaved pipeline scheduling process. Figure 3 It is an interleaved pipeline scheduling process. Figure 2 and Figure 3 The pipeline scheduling process shown includes 8 micro-batches. Each micro-batch can include part of the training data. Each micro-batch needs to be processed sequentially in the forward and reverse directions by four execution devices to achieve model training for the pipeline. During the sequential execution of each micro-batch, after the previous execution device completes its calculation for the batch, it needs to send the calculation result to the next execution device so that the next execution device can perform subsequent calculations based on the calculation result of the previous execution device.

[0039] Before the execution device sends the tensor parameter to other execution devices, the execution device may detect the tensor parameter to determine whether the tensor parameter contains abnormal data. If the tensor parameter contains abnormal data, the execution device may generate abnormal information based on the abnormal data and send the abnormal information to the control device. The abnormal information is used to characterize the occurrence of training abnormalities in the execution device. Optionally, the abnormal information may include information such as the device ID of the execution device, the tensor name or identifier of the tensor parameter, the abnormal type, and a timestamp. Optionally, the execution device may include multiple tensor parameters to be detected. When there is a tensor among the multiple tensor parameters that contains abnormal data, the execution device generates abnormal information.

[0040] Optionally, if none of the multiple tensor parameters contain abnormal data, the execution device can output the tensor parameter according to the other execution devices to which the tensor parameter needs to be output. Optionally, the multiple tensor parameters can be output to multiple other execution devices. Optionally, when there are multiple tensor parameters that need to be output to the same execution device, the multiple tensor parameters can be merged first. The execution device can output the merged multiple tensor parameters together to the corresponding execution device. The merged transmission of the parameters improves the efficiency of data transmission between execution devices and improves the training efficiency of the model.

[0041] Optionally, the abnormal data may be one or more of the following: data in a preset format, data with abnormal precision, and data with abnormal variations. Data in a preset format may include data such as NaN and Inf. Data with abnormal precision is data with a value outside the precision range. Data with abnormal variations is data with an iterative difference that does not conform to a preset difference range.

[0042] Optionally, the tensor parameters may include one or more of the following: forward tensor parameters, reverse tensor parameters, parameter gradients, and model parameters. The forward tensor parameters are the forward propagation calculation results obtained by the execution device, which are forward-outputted to the next execution device by the executed device according to the pipeline direction. The reverse tensor parameters are the reverse propagation calculation results obtained by the execution device, which are reverse-outputted to the previous execution device by the executed device according to the pipeline direction. The gradient parameters are used to indicate the gradient of change during model parameter training. Model parameters are the trained parameters of the model to be trained.

[0043] 102. Control the execution device to stop executing the training process.

[0044] In this embodiment, when the execution device detects abnormal data, the execution device can be controlled to stop the execution of the current training process. Optionally, the execution device can stop sending the current tensor parameters.

[0045] In this embodiment, by detecting whether there is abnormal data in the tensor parameters output by the execution device, the execution device can promptly alarm, locate, and stop model training when abnormal data occurs, thereby avoiding further expansion of the abnormal data and causing abnormal model parameters to become abnormal, and can promptly locate the execution device where the abnormality occurs, thereby improving the efficiency of subsequent exception handling.

[0046] Figure 4 is a schematic diagram according to the second embodiment of the present disclosure, as shown in Figure 4As shown, the present disclosure provides an anomaly detection method based on a pipeline parallel training strategy. The method is used to detect abnormal data in tensor parameters. The abnormal data may include data in a preset format, data with abnormal accuracy, data with abnormal changes, etc. Based on these anomalies, the specific implementation of anomaly detection may include the following steps:

[0047] 401. If data in a preset format is detected in the data of the tensor parameter, it is determined that abnormal data is detected.

[0048] In this embodiment, a tensor detection function can be preset in the execution device. The tensor detection function can be used to detect data in a preset format in the tensor parameter. Optionally, the preset format can be NaN, Inf, etc. Optionally, the tensor detection function can be a function such as isnan() and isinf(). Optionally, the tensor detection function can be a function in an existing deep learning framework, or the tensor detection function can be a function customized as needed. The detection of abnormal data in the preset format can prevent the data in the abnormal format from being propagated during the model training process, resulting in abnormal model parameters.

[0049] 402. If it is detected that the data of the tensor parameter contains data that exceeds the precision abnormality, it is determined that abnormal data is detected.

[0050] In this embodiment, a tensor detection function may be preset in the execution device. The tensor detection function may also be used to detect whether the data in the tensor parameter has an accuracy anomaly. If data with an accuracy anomaly exists, it is determined that abnormal data has been detected.

[0051] Optionally, the process of detecting data with abnormal accuracy may include the following two steps:

[0052] 4021. Determine an accuracy range based on the current convergence level information of the model to be trained.

[0053] 4022. If it is detected that data in the tensor parameter is out of the accuracy range, determine the data in the tensor parameter that has accuracy anomalies.

[0054] In this embodiment, the execution device may be pre-set with a precision range for precision anomaly detection. The tensor detection function may be used to check whether the data in the tensor parameter is within the precision range. If the data is within the precision range, the numerical value is determined to meet the precision requirements. If the value is not within the precision range, the numerical value is determined to meet the precision requirements and a precision anomaly exists. If data with a precision anomaly exists, it is determined that abnormal data has been detected.

[0055] Optionally, the accuracy range can be determined based on the current convergence degree information of the model to be trained. For example, an initial numerical range can be preset in the execution device. When the model just starts to be trained, the initial numerical range can be used as the accuracy range. As the model is trained, the convergence degree of the model changes. According to the convergence degree of the model in the current iteration, the initial data range can be narrowed to obtain a smaller numerical range, and the smaller numerical range is used as the accuracy range. Optionally, the narrowing method can be to determine a reduction ratio based on the convergence degree. The new numerical range is determined based on the product of the reduction ratio and the initial data range. The dynamic adjustment of the accuracy range can further ensure the validity of the data during the model training process, avoid numerical jumps or the appearance of singular values, which may cause the accuracy of the model to be abnormal in subsequent training.

[0056] Optionally, if the model to be trained is a model for speech recognition, the tensor parameters are speech feature parameters in the model used for speech recognition. The accuracy range can also be adjusted based on a preset frequency range of human voice, thereby further improving the validity of the data and allowing the model to be processed promptly when invalid or abnormal data is found.

[0057] 403. If the model to be trained is a model for image recognition, and the tensor parameters are image feature parameters in the model for image recognition, gradient information between pixels in the tensor parameters is obtained. If an abnormal change in the gradient information is detected, abnormal data is determined to have been detected.

[0058] In this embodiment, if the model to be trained is a model used for image recognition, the tensor parameters are image feature parameters in the model used for image recognition. Based on the image feature parameters, gradient information between pixels can be obtained, and the gradients can be detected to determine whether there are abnormal changes in the gradient information of each pixel. If there are abnormal changes, it is determined that abnormal data has been detected.

[0059] Optionally, the abnormal change can specifically be whether the direction indicated by the gradient information is consistent. If so, the pixel is normal. If not, the pixel has undergone an abnormal change and can be treated as abnormal data. This detection method can be based on an image recognition model, further improving model processing efficiency and avoiding excessive image processing during model training, which can lead to poor model training results.

[0060] 404. Detecting the iteration difference of the tensor parameters of the training model during the training process. The iteration difference represents the data difference between the previous iteration and the current iteration of the tensor parameters during the training process. If it is determined whether the tensor parameters have abnormal changes, it is determined that abnormal data has been detected.

[0061] In this embodiment, the execution device may also store tensor parameters from the previous iteration. After calculating the tensor parameters for the current iteration, the difference between the tensor parameters for the local iteration and the tensor parameters for the previous iteration may be calculated. Based on the difference in the tensor parameters, it may be determined whether there has been an abnormal change in the tensor parameters. If there has been an abnormal change, it is determined that abnormal data has been detected.

[0062] Optionally, the detection of whether there is an abnormal change can be achieved by presetting a difference range. If the difference is within the preset difference range, then the tensor parameter has not undergone an abnormal change during the iteration. If the difference is not within the preset range, then the tensor parameter has undergone an abnormal change during the iteration, and it is determined that a data anomaly has been detected. This anomaly detection process can further ensure the stability of the tensor parameters during the iteration process, thereby improving the effectiveness of model training.

[0063] In this embodiment, multiple anomaly detection methods are used to detect multiple anomalies in the tensor parameters, thereby improving the accuracy of anomaly detection of tensor parameters and ensuring the efficiency and effect of model training.

[0064] On the basis of the above embodiment, the execution device can also send the model parameters of the model training team generated by the iteration to the control device for storage after completing one iteration of training. Optionally, the execution device can send the model parameters after each iteration. Or the execution device can send them periodically according to a fixed cycle. For example, the execution device can send the model parameters once every five iterations. The sending of the model parameters can synchronize the model parameters to the replacement execution device when an abnormality occurs in the execution device, so that the replacement execution device can take over the abnormal execution device for subsequent training.

[0065] Figure 5 is a schematic diagram according to the third embodiment of the present disclosure, as shown in Figure 5 As shown, the present disclosure provides an anomaly detection method based on a pipeline parallel training strategy. In the process of pipeline parallel training, the execution steps of the execution device and the anomaly detection steps during the execution process can be as follows, including:

[0066] 501. Forward calculation.

[0067] In this embodiment, on each execution device, the execution device may perform forward propagation calculation on the input data to obtain a forward tensor parameter. The forward tensor parameter may specifically be a forward activation value.

[0068] 502. Check the forward tensor parameters.

[0069] In this embodiment, by performing an anomaly check on the forward tensor parameters, it is determined whether there is abnormal data in the forward tensor, thereby ensuring the validity of the forward calculation results and preventing the generation and propagation of abnormal data. Optionally, the anomaly check includes at least a check for NaN or Inf values.

[0070] Optionally, the check for abnormal values ​​can be implemented by writing an efficient tensor check function. The tensor check function can quickly detect whether there is abnormal data in the tensor parameter. For example, abnormality detection for NaN or Inf values ​​can be implemented through the isnan() and isinf() functions. Optionally, the function can be a custom function, or the function can be a function in the PyTorch or TensorFlow deep learning framework. Optionally, the abnormality detection can be implemented in the distributed model training environment of Paddlepaddle.

[0071] Optionally, when abnormal data is detected, the execution device can immediately interrupt the current calculation and generate an exception message. The exception message may include at least the device ID, which is used to identify the execution device where the exception occurred, the timestamp of the exception occurrence, the name or identifier of the tensor where the exception occurred, and the exception type of NaN or Inf.

[0072] Optionally, the execution device may report the abnormal information to a control device. Optionally, the control device may be a central monitoring or logging system. The control device may perform fault diagnosis based on the abnormal information.

[0073] 503. Send forward tensor parameters.

[0074] In this embodiment, the execution device can send the forward calculation tensor that has been checked and confirmed to be normal to the next device to continue the pipeline calculation. Optionally, the next device can be the next execution device of the execution device on the pipeline.

[0075] 504. Reverse calculation.

[0076] In this embodiment, the execution device may calculate the back propagation of the error to obtain a reverse tensor parameter. Optionally, the reverse tensor parameter includes gradient information.

[0077] 505. Check the reverse tensor parameters.

[0078] In this embodiment, to ensure the validity of the reverse tensor parameters calculated in reverse and prevent abnormal gradients in the reverse tensor parameters from causing parameter update errors, the reverse tensor parameters can be checked for abnormal data. The abnormal data detection process and the handling method when abnormal data is found are the same as the calculation process of the forward tensor parameters described above.

[0079] 506. Send reverse tensor parameters.

[0080] In this embodiment, the execution device sends the checked reverse tensor parameters to the next device. Optionally, the next device may be the previous execution device of the execution device in the pipeline.

[0081] 507. Circular judgment.

[0082] In this embodiment, if the forward and reverse calculations of each execution device on the pipeline have been completed, the subsequent steps are continued to be executed. Otherwise, the process enters a waiting state until the forward and reverse calculations of each execution device are completed.

[0083] 508. Check parameter gradients.

[0084] In this embodiment, the execution device can ensure the validity of all gradient values ​​before updating the model parameters. Optionally, the execution device can detect abnormal data in the local training parameter gradients. This detection process and the handling of abnormal data are the same as those for the forward tensor parameters.

[0085] 509. Cross-device gradient communication.

[0086] In this embodiment, each execution device may communicate parameter gradients to achieve synchronization of the parameter gradients on each execution device, in preparation for updating the model parameters.

[0087] 510. Optimizer update.

[0088] In this embodiment, the execution device may use the synchronized parameter gradients to update the model parameters according to the optimizer algorithm.

[0089] In this embodiment, through the above process, the execution of the execution device in the parallel training process of the pipeline is realized, and the abnormal data of the tensor parameters is detected during the training process, so that when an abnormality occurs during the model training process, the abnormality can be located and processed in time to avoid the impact of the abnormality on the model training.

[0090] Figure 6 is a schematic diagram according to a fourth embodiment of the present disclosure, as shown in Figure 6 As shown, the present disclosure provides an anomaly detection device 600 based on a pipeline parallel training strategy, the device comprising:

[0091] Detection unit 610 is configured to, during a training process of a model to be trained based on an execution device in a device set, send an exception message to a control device if abnormal data is detected in a tensor parameter of the execution device during training. The device set includes multiple execution devices. The exception message indicates that a training anomaly occurred on the execution device.

[0092] The control unit 620 is used to control the execution device to stop executing the training process.

[0093] The device of this embodiment can execute the technical solution in the above method. Its specific implementation process and technical principles are the same and will not be repeated here.

[0094] Figure 7 is a schematic diagram according to a fifth embodiment of the present disclosure, as shown in Figure 7 As shown, the present disclosure provides an anomaly detection device 700 based on a pipeline parallel training strategy, the device comprising:

[0095] A detection unit 710 is configured to, during a training process of a to-be-trained model based on an execution device in a device set, send an exception message to a control device if abnormal data is detected in a tensor parameter of the execution device during training. The device set may include multiple execution devices. The exception message indicates that a training anomaly occurred on the execution device.

[0096] The control unit 720 is used to control the execution device to stop executing the training process.

[0097] Optionally, the abnormal data is one or more of the following: data in a preset format, data with abnormal accuracy, and data with abnormal changes.

[0098] Optionally, the detection unit 710 includes:

[0099] The first detection module 711 is configured to determine that abnormal data has been detected if data having a preset format is detected in the data of the tensor parameter based on a preset tensor detection function.

[0100] Optionally, the detection unit 710 includes:

[0101] The second detection module 712 is configured to determine that abnormal data has been detected if, based on a preset tensor detection function, it is detected that the data of the tensor parameters contain data that exceeds a precision range.

[0102] Optionally, the second detection module 712 includes:

[0103] The first parameter submodule 7121 is used to determine the accuracy range according to the current convergence degree information of the model to be trained.

[0104] Optionally, if the model to be trained is a model applied to speech recognition, the tensor parameters are speech feature parameters in the model applied to speech recognition. The second detection module 712 includes:

[0105] The second parameter submodule 7122 is used to adjust the accuracy range according to the preset frequency range of human voice.

[0106] Optionally, if the model to be trained is a model applied to image recognition, the tensor parameters are image feature parameters in the model applied to image recognition. The detection unit 710 includes:

[0107] The third detection module 713 is used to obtain gradient information between pixels in the tensor parameter. If it is detected that the directions indicated by the gradient information are inconsistent, it is determined that abnormal data has been detected.

[0108] Optionally, the detection unit 710 includes:

[0109] The fourth detection module 714 is configured to detect the iterative difference of tensor parameters during the training process of the training model. The iterative difference represents the difference between the data of the previous iteration and the current iteration of the tensor parameters during the training process. If the iterative difference of the tensor parameters is determined to be outside the preset difference range, abnormal data is detected.

[0110] Optionally, tensor parameters include one or more of the following: forward tensor parameters, reverse tensor parameters, parameter gradients, and model parameters. Forward tensor parameters are the forward propagation calculation results obtained by the execution device, which are forward-outputted to the next execution device by the executed device according to the pipeline direction. Reverse tensor parameters are the backward propagation calculation results obtained by the execution device, which are reverse-outputted to the previous execution device by the executed device according to the pipeline direction. Gradient parameters are used to indicate the gradient of change during model parameter training. Model parameters are the trained parameters of the model to be trained.

[0111] Optionally, the anomaly detection device 700 further includes:

[0112] The sending unit 730 is used to send the model parameters updated during the training process of the model to be trained to the management and control device for storage.

[0113] Optionally, the anomaly detection device 700 further includes:

[0114] The merging unit 740 is used to merge multiple tensor parameters output to the same execution device during the training process of training the to-be-trained model based on the execution device in the device set, if no abnormal data is detected in the tensor parameters of the execution device during the training process, and transmit the merged multiple tensor parameters.

[0115] The device of this embodiment can execute the technical solution in the above method. Its specific implementation process and technical principles are the same and will not be repeated here.

[0116] Figure 8 is a schematic diagram according to a sixth embodiment of the present disclosure, as shown in Figure 8As shown, the present disclosure provides an anomaly detection system 800 based on a pipeline parallel training strategy, the system comprising:

[0117] The device set 810 and the control device 820 include multiple execution devices 811. The execution device is used to execute the above Figures 1 to 3 In the illustrated embodiment, an anomaly detection method based on a pipelined parallel training strategy is employed. The control device determines a replacement execution device from among idle execution devices in a device set based on anomaly information. The control device then transmits the model parameters of the execution device with the anomaly information to the candidate execution device, allowing the replacement execution device to participate in the training process in place of the execution device with the anomaly information based on the model parameters.

[0118] In this embodiment, the management and control device schedules the execution device with abnormal information and the replacement execution device, thereby further ensuring the training efficiency of the model, so that the model training will not be affected by abnormal data, and improving the efficiency of abnormal processing.

[0119] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0120] According to an embodiment of the present disclosure, the present disclosure also provides a computer program product, which includes: a computer program, the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device executes the solution provided by any of the above embodiments.

[0121] Figure 9 A schematic block diagram of an example electronic device 900 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0122] like Figure 9As shown, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. Various programs and data required for the operation of the device 900 can also be stored in the RAM 903. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0123] Various components in the device 900 are connected to the I / O interface 905, including an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a magnetic disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0124] The computing unit 901 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 901 performs the various methods and processes described above, such as the anomaly detection method based on the pipeline parallel training strategy. For example, in some embodiments, the anomaly detection method based on the pipeline parallel training strategy can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 908. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the anomaly detection method based on the pipeline parallel training strategy described above can be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to execute the anomaly detection method based on the pipeline parallel training strategy in any other appropriate manner (for example, by means of firmware).

[0125] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0126] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0127] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0128] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0129] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0130] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact via a communication network. This client-server relationship is established by computer programs running on the respective computers, establishing a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host, a host product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosts and VPS services ("Virtual Private Servers" or simply "VPS"). The server may also be a server in a distributed system or a server integrated with blockchain.

[0131] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not a limitation herein.

[0132] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. An anomaly detection method based on a pipeline parallel training strategy, comprising: During a training process of a to-be-trained model being trained by an execution device in a device set, if abnormal data is detected in a tensor parameter of the execution device during the training process, an abnormality message is sent to a control device; wherein the device set includes a plurality of execution devices; and the abnormality message indicates that a training abnormality has occurred in the execution device; Controlling the execution device to stop executing the training process; The tensor parameters include one or more of the following: forward tensor parameters, reverse tensor parameters, gradient parameters, and model parameters; Among them, the forward tensor parameter is the calculation result of the forward propagation calculated by the execution device, which will be output forward to the next execution device by the execution device according to the pipeline direction; the reverse tensor parameter is the calculation result of the reverse propagation calculated by the execution device, which will be output reversely to the previous execution device by the execution device according to the pipeline direction; the gradient parameter is used to indicate the change gradient of the model parameter training process; the model parameter is the trained parameter in the model to be trained.

2. The method according to claim 1, wherein the abnormal data comprises data in a preset format; and detecting that the tensor parameters of the execution device during training have abnormal data comprises: If data in a preset format is detected in the data of the tensor parameter, it is determined that the abnormal data is detected.

3. The method according to claim 1, wherein the abnormal data comprises data with abnormal accuracy; and detecting that the tensor parameters of the execution device during training have abnormal data comprises: If data with abnormal accuracy is detected in the data of the tensor parameters, it is determined that the abnormal data is detected.

4. The method according to claim 3, wherein the detecting of data with abnormal accuracy in the data of the tensor parameters comprises: Determining an accuracy range based on current convergence degree information of the model to be trained; If it is detected that there is data in the tensor parameter that is not within the precision range, it is determined that the data of the tensor parameter has precision abnormality.

5. The method according to claim 4, wherein if the model to be trained is a model applied to speech recognition, the tensor parameters are speech feature parameters in the model applied to speech recognition; the method further comprises: The accuracy range is adjusted according to a preset frequency range of human voice.

6. The method according to claim 1, wherein the abnormal data comprises data with abnormal changes; if the model to be trained is a model applied to image recognition, the tensor parameters are image feature parameters in the model applied to image recognition; Detecting that a tensor parameter of the execution device during training has abnormal data includes: Obtaining gradient information between pixels in the tensor parameter; If an abnormal change in the gradient information is detected, it is determined that the abnormal data is detected.

7. The method according to claim 1, wherein the abnormal data comprises data with abnormal changes; Detecting that a tensor parameter of the execution device during training has abnormal data includes: Detecting an iterative difference of a tensor parameter of the to-be-trained model during a training process; wherein the iterative difference represents a data difference between a previous iteration and a current iteration of the tensor parameter during the training process; If it is determined that there is an abnormal change in the iterative difference of the tensor parameter, it is determined that the abnormal data is detected.

8. The method according to any one of claims 1 to 7, further comprising: During the training process of the model to be trained, the model parameters updated during the training process are sent to the control device for storage.

9. The method according to any one of claims 1 to 7, further comprising: During the training process of training a model to be trained based on an execution device in a device set, if no abnormal data is detected in the tensor parameters of the execution device during the training process, multiple tensor parameters output to the same execution device will be merged and the merged multiple tensor parameters will be transmitted.

10. An anomaly detection device based on a pipeline parallel training strategy, comprising: a detection unit configured to, during a training process of a to-be-trained model being trained by an execution device in a device set, send an abnormality message to a control device if abnormal data is detected in a tensor parameter of the execution device during training; wherein the device set includes a plurality of execution devices; and the abnormality message indicates that a training abnormality has occurred in the execution device; a control unit, configured to control the execution device to stop executing the training process; The tensor parameters include one or more of the following: forward tensor parameters, reverse tensor parameters, gradient parameters, and model parameters; Among them, the forward tensor parameter is the calculation result of the forward propagation calculated by the execution device, which will be output forward to the next execution device by the execution device according to the pipeline direction; the reverse tensor parameter is the calculation result of the reverse propagation calculated by the execution device, which will be output reversely to the previous execution device by the execution device according to the pipeline direction; the gradient parameter is used to indicate the change gradient of the model parameter training process; the model parameter is the trained parameter in the model to be trained.

11. An anomaly detection system based on a pipeline parallel training strategy, the system comprising a device set and a control device, the device set comprising a plurality of execution devices; wherein, The execution device is used to execute the method according to any one of claims 1 to 9.

12. According to the system according to claim 11, the management and control device is used to determine a replacement execution device from the execution devices in the device set that are in an idle state based on the exception information; and send model parameters of the execution device with exception information to the replacement execution device, so that the replacement execution device takes over the execution device with exception information to participate in the training process based on the model parameters.

13. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 9.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 9.

15. A computer program product comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 9.

Images