Abnormality processing method and device, electronic equipment and storage medium

Abstract

The invention discloses an exception handling method and device, electronic equipment and a storage medium, and the method comprises the steps: detecting the computing power exception information of a heterogeneous computing equipment cluster in the training of a deep learning model; if so, interrupting the training of the deep learning model on the heterogeneous computing device cluster; searching a target task processing strategy matched with the abnormal computing power index from a current task processing strategy set; task processing strategies in the current task processing strategy set correspond to different computing power indexes, and each task processing strategy represents training resources needing to be loaded under the condition that the corresponding computing power index is abnormal; and if the abnormal heterogeneous computing device is found, restarting training of the deep learning model on a target heterogeneous computing device cluster based on the target task processing strategy, wherein the target heterogeneous computing device cluster comprises heterogeneous computing devices except the abnormal heterogeneous computing device. The method does not need manual follow-up to solve breakpoint continuous training of each time, shortens the training time, and improves the utilization rate of cluster resources.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an exception handling method, device, electronic device and storage medium. Background Art

[0002] With the rapid development of artificial intelligence (AI) technology, large-scale deep learning models are being used more and more, such as large-scale natural language processing (NLP) models and large-scale image processing models.

[0003] Generally, large-scale deep learning models have complex network structures and large number of parameters. When executing the training of large-scale deep learning models, the scale of data exchanged during the training process is also high, and the scale of GPU (Graphics Processing Unit) computing power required will reach more than 10,000 cards. In the related technology, in the process of training deep learning models on such a large-scale GPU computing power device cluster, every time the cluster computing power is abnormal, manual follow-up and resolution are required, resulting in a long training time for deep learning models and a waste of GPU computing power resources. Summary of the invention

[0004] In order to solve the problems of the prior art, the embodiments of the present application provide an exception handling method, device, electronic device and storage medium. The technical solution is as follows:

[0005] In one aspect, a method for handling an exception is provided, the method comprising:

[0006] In the process of training a deep learning model based on a heterogeneous computing device cluster, abnormal computing power information of the heterogeneous computing device cluster is detected; the abnormal computing power information represents an abnormal heterogeneous computing device with abnormal computing power indicators, and an abnormal computing power indicator corresponding to the abnormal heterogeneous computing device;

[0007] If the computing power abnormality information is detected, interrupting the training of the deep learning model on the heterogeneous computing device cluster;

[0008] From the current task processing strategy set, search for a target task processing strategy that matches the abnormal computing power indicator; the task processing strategies in the current task processing strategy set correspond to different computing power indicators, each task processing strategy represents a training resource that needs to be loaded when an abnormality occurs in the corresponding computing power indicator, and the training resource includes the training status information corresponding to the deep learning model when the training is interrupted;

[0009] If the target task processing strategy is found, the training of the deep learning model on the target heterogeneous computing device cluster is restarted based on the target task processing strategy; the target heterogeneous computing device cluster includes heterogeneous computing devices except the abnormal heterogeneous computing device.

[0010] On the other hand, a method for handling an exception is also provided, the method comprising:

[0011] In the process of language processing training of a natural language processing model based on a heterogeneous computing device cluster, abnormal computing power information of the heterogeneous computing device cluster is detected; the abnormal computing power information represents an abnormal heterogeneous computing device with abnormal computing power indicators, and an abnormal computing power indicator corresponding to the abnormal heterogeneous computing device;

[0012] If the computing power abnormality information is detected, interrupting the language processing training of the natural language processing model on the heterogeneous computing device cluster;

[0013] From the current task processing strategy set, searching for a target task processing strategy that matches the abnormal computing power indicator; the task processing strategies in the current task processing strategy set correspond to different computing power indicators, and each task processing strategy represents a training resource that needs to be loaded when an abnormality occurs in the corresponding computing power indicator; the training resource includes the training status information corresponding to the natural language processing model when the language processing training is interrupted;

[0014] If the target task processing strategy is found, the language processing training of the natural language processing model on the target heterogeneous computing device cluster is restarted based on the target task processing strategy; the target heterogeneous computing device cluster includes heterogeneous computing devices except the abnormal heterogeneous computing device.

[0015] In another aspect, an exception handling device is provided, the device comprising:

[0016] A first computing power anomaly detection module is used to detect computing power anomaly information of a heterogeneous computing device cluster during the process of training a deep learning model based on the heterogeneous computing device cluster; the computing power anomaly information represents an abnormal heterogeneous computing device with abnormal computing power indicators, and an abnormal computing power indicator corresponding to the abnormal heterogeneous computing device;

[0017] A first training interruption module, configured to interrupt the training of the deep learning model on the heterogeneous computing device cluster when the computing power abnormality information is detected;

[0018] A first processing strategy search module is used to search for a target task processing strategy that matches the abnormal computing power indicator from the current task processing strategy set; the task processing strategies in the current task processing strategy set correspond to different computing power indicators, each task processing strategy represents a training resource that needs to be loaded when an abnormality occurs in the corresponding computing power indicator, and the training resource includes the training state information corresponding to the deep learning model when the training is interrupted;

[0019] The first training restart module is used to restart the training of the deep learning model on the target heterogeneous computing device cluster based on the target task processing strategy when the target task processing strategy is found; the target heterogeneous computing device cluster includes heterogeneous computing devices other than the abnormal heterogeneous computing device.

[0020] In some exemplary embodiments, the apparatus further comprises:

[0021] A first task processing configuration module is used to obtain task processing configuration information input for the abnormal computing power indicator based on a task processing configuration interface when the target task processing strategy is not found; the task processing configuration information represents the training resources that need to be loaded when the abnormal computing power indicator appears;

[0022] A third training restart module, configured to restart the training of the deep learning model on the target heterogeneous computing device cluster based on the task processing configuration information;

[0023] The first task processing strategy set updating module is used to use the task processing configuration information as the task processing strategy in the current task processing strategy set to update the current task processing strategy set.

[0024] In some exemplary embodiments, the computing power indicator includes at least one functional indicator, each of which represents a preset operating state of the hardware card; the first computing power anomaly detection module includes:

[0025] A first detection module is configured to detect, for each heterogeneous computing device in the heterogeneous computing device cluster, at least one of the preset operating states of the hardware card in the heterogeneous computing device, and obtain a state detection result of the heterogeneous computing device corresponding to each of the preset operating states; the state detection result indicates whether the hardware card is in the corresponding preset operating state;

[0026] A first determination module is configured to determine that an abnormality exists in a target functional indicator of the heterogeneous computing device when a first state detection result exists in the state detection results corresponding to each of the preset operating states of the heterogeneous computing device; the first state detection result indicates that the hardware card has a first preset operating state corresponding to the first state detection result, and the target functional indicator is a functional indicator corresponding to the first preset operating state;

[0027] The first generating module is used to generate computing power abnormality information of the heterogeneous computing device cluster based on the address information of the heterogeneous computing device and the target function indicator.

[0028] In some exemplary embodiments, the computing power indicator includes at least one performance indicator; and the first computing power anomaly detection module includes:

[0029] A data acquisition module, configured to acquire, for each heterogeneous computing device in the heterogeneous computing device cluster, performance indicator data corresponding to each performance indicator of the heterogeneous computing device;

[0030] A second determination module, configured to determine a fluctuation range of each performance indicator of the heterogeneous computing device according to performance indicator data of each performance indicator of the heterogeneous computing device;

[0031] The second detection module is used to compare the fluctuation range of each performance indicator of the heterogeneous computing device with the preset fluctuation range threshold corresponding to each performance indicator, and obtain the performance detection result of each performance indicator of the heterogeneous computing device; the performance detection result indicates whether the fluctuation range of the corresponding performance indicator exceeds the preset fluctuation range threshold corresponding to the performance indicator;

[0032] A third determination module is configured to determine that a target performance indicator of the heterogeneous computing device is abnormal when a first performance detection result exists in the performance detection results corresponding to each performance indicator of the heterogeneous computing device; the target performance indicator is a performance indicator corresponding to the first performance detection result, and the first performance detection result indicates that a fluctuation range of the target performance indicator exceeds a preset fluctuation range threshold corresponding to the target performance indicator;

[0033] The second generating module is used to generate computing power anomaly information of the heterogeneous computing device cluster based on the address information of the heterogeneous computing device and the target performance indicator.

[0034] In some exemplary embodiments, the apparatus further comprises:

[0035] An abnormal marking module, used for marking the abnormal heterogeneous computing device in the heterogeneous computing device cluster based on the computing power abnormality information;

[0036] The fourth determination module is used to determine the cluster composed of unmarked heterogeneous computing devices in the heterogeneous computing device cluster as a target heterogeneous computing device cluster.

[0037] In some exemplary embodiments, the apparatus further comprises:

[0038] A first training state recording module is used to record the training state of the deep learning model at preset time intervals during the training process of the deep learning model to obtain training state information corresponding to each time point;

[0039] The first training restart module includes:

[0040] A fifth determination module is used to determine a target time point that is closest to interrupting the training among the time points, and use the training state information corresponding to the target time point as the training state information corresponding to the training resource in the target task processing strategy;

[0041] The first training restart submodule is used to restart the training of the deep learning model on the target heterogeneous computing device cluster based on the target task processing strategy.

[0042] On the other hand, an exception handling device is also provided, the device comprising:

[0043] A second computing power anomaly detection module is used to detect computing power anomaly information of the heterogeneous computing device cluster during the process of language processing training of the natural language processing model based on the heterogeneous computing device cluster; the computing power anomaly information represents an abnormal heterogeneous computing device with abnormal computing power indicators, and an abnormal computing power indicator corresponding to the abnormal heterogeneous computing device;

[0044] A second training interruption module is used to interrupt the language processing training of the natural language processing model on the heterogeneous computing device cluster when the computing power abnormality information is detected;

[0045] A second processing strategy search module is used to search for a target task processing strategy that matches the abnormal computing power indicator from the current task processing strategy set; the task processing strategies in the current task processing strategy set correspond to different computing power indicators, and each task processing strategy represents a training resource that needs to be loaded when the corresponding computing power indicator is abnormal; the training resource includes the training state information corresponding to the natural language processing model when the language processing training is interrupted;

[0046] The second training restart module is used to restart the language processing training of the natural language processing model on the target heterogeneous computing device cluster based on the target task processing strategy when the target task processing strategy is found; the target heterogeneous computing device cluster includes heterogeneous computing devices except the abnormal heterogeneous computing device.

[0047] In some exemplary embodiments, the apparatus further comprises:

[0048] A second task processing configuration module is used to obtain task processing configuration information input for the abnormal computing power indicator based on the task processing configuration interface when the target task processing strategy is not found; the task processing configuration information represents the training resources that need to be loaded when the abnormal computing power indicator appears;

[0049] a fourth training restart module, configured to restart the language processing training of the natural language processing model on the target heterogeneous computing device cluster based on the task processing configuration information;

[0050] The second task processing strategy set updating module is used to use the task processing configuration information as the task processing strategy in the current task processing strategy set to update the current task processing strategy set.

[0051] In some exemplary embodiments, the apparatus further comprises:

[0052] A second training state recording module is used to record the training state of the natural language processing model at preset time intervals during the language processing training process of the natural language processing model to obtain training state information corresponding to each time point;

[0053] The second training restart module includes:

[0054] A sixth determination module is used to determine a target time point that is closest to interrupting the language processing training among the time points, and use the training state information corresponding to the target time point as the training state information corresponding to the training resource in the target task processing strategy;

[0055] The second training restart submodule is used to restart the language processing training of the natural language processing model on the target heterogeneous computing device cluster based on the target task processing strategy.

[0056] On the other hand, an electronic device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement any of the above-mentioned exception handling methods.

[0057] On the other hand, a computer-readable storage medium is provided, in which at least one instruction or at least one program is stored, and the at least one instruction or the at least one program is loaded and executed by a processor to implement the exception handling method as described in any of the above aspects.

[0058] On the other hand, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device performs any of the above-mentioned exception handling methods.

[0059] The embodiment of the present application detects abnormal computing power information of a heterogeneous computing device cluster during the training process of a deep learning model, and interrupts the training of the deep learning model on the heterogeneous computing device cluster when the abnormal computing power information is detected, and then searches for a target task processing strategy that matches the abnormal computing power index from the current task processing strategy set, and restarts the training of the deep learning model on the target heterogeneous computing device cluster based on the target task processing strategy when the target task processing strategy is found, wherein the task processing strategies in the current task processing strategy set correspond to different computing power indicators, each task processing strategy represents the training resources that need to be loaded when the corresponding computing power indicator is abnormal, and the training resources include the training state information corresponding to the deep learning model when the training is interrupted, and the target heterogeneous computing device cluster includes heterogeneous computing devices other than the abnormal heterogeneous computing device. Thus, the above technical solution realizes the automatic continuation of breakpoint training, without the need for manual follow-up to solve each breakpoint continuation training, greatly shortens the training time of the deep learning model, improves the utilization rate of the heterogeneous computing device cluster, and saves cluster computing power resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0061] Figure 1 It is a schematic diagram of an implementation environment provided by an embodiment of the present application;

[0062] Figure 2 It is a flowchart of an exception handling method provided in an embodiment of the present application;

[0063] Figure 3 It is a flowchart of another exception handling method provided in an embodiment of the present application;

[0064] Figure 4 It is a flowchart of another exception handling method provided in an embodiment of the present application;

[0065] Figure 5 It is a flowchart of another exception handling method provided in an embodiment of the present application;

[0066] Figure 6 This is an example of an architecture of an exception handling method provided in an embodiment of the present application;

[0067] Figure 7 It is a flowchart of another exception handling method provided in an embodiment of the present application;

[0068] Figure 8 is a structural block diagram of an exception handling device provided in an embodiment of the present application;

[0069] Fig. 9 is a structural block diagram of another exception handling device provided in an embodiment of the present application;

[0070] Fig.10 It is a hardware structure block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0071] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0072] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0073] It is understandable that in the specific implementation of this application, related data such as user information is involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0074] See also Figure 1 , which is a schematic diagram of an implementation environment provided by an embodiment of the present application, wherein the implementation environment includes a heterogeneous computing device cluster 110 and a server 120.

[0075] Among them, the heterogeneous computing device cluster 110 includes multiple heterogeneous computing devices. When training large-scale deep learning models such as natural language processing models, the multiple heterogeneous computing devices form a large computing network. Training data can be exchanged between multiple devices in the computing network. After the training data is exchanged to the heterogeneous computing devices, the heterogeneous computing devices will perform model training calculations, and the calculated intermediate result data will be exchanged as training data to other heterogeneous computing devices.

[0076] Among them, heterogeneous computing devices have the characteristics of large computing power, and can also be called heterogeneous artificial intelligence acceleration hardware. Heterogeneous computing devices may include but are not limited to graphics processing units (GPUs), neural network processors (Neural-network Process Units, NPUs), tensor processors (Tensor Processing Unit, TPUs), FPGA (Field Programmable Gate Array, Field Programmable Logic Gate Array) hardware, etc. It should be noted that the heterogeneous computing devices in the heterogeneous computing device cluster 110 may include heterogeneous computing devices of different manufacturers and different types, such as GPUs, NPUs, and TPUs from multiple different manufacturers.

[0077] The server 120 is used to detect the abnormal computing power information of the heterogeneous computing device cluster 110 during the training process of the deep learning model, and interrupt the training of the deep learning model on the heterogeneous computing device cluster 110 when the abnormal computing power information is detected, and then find the target task processing strategy that matches the abnormal computing power index indicated by the abnormal computing power information from the current task processing strategy set, and restart the training of the deep learning model on the target heterogeneous computing device cluster based on the target task processing strategy when the target task processing strategy is found, wherein the task processing strategies in the current task processing strategy set correspond to different computing power indicators, and each task processing strategy represents the training resources that need to be loaded when the corresponding computing power indicator is abnormal, and the training resources include the training state corresponding to the deep learning model when the training is interrupted, and the target heterogeneous computing device cluster includes heterogeneous computing devices other than the abnormal heterogeneous computing devices indicated by the abnormal computing power information. Thus, the above technical solution realizes the automatic continuation of breakpoint training, without the need for manual follow-up to solve each breakpoint continuation training, greatly shortens the training time of the deep learning model, improves the utilization rate of heterogeneous computing devices, and saves cluster computing power resources.

[0078] In some specific implementation scenarios, the server 120 may include a computing power detection server and a training server. The computing power detection server is used to detect computing power anomalies on the heterogeneous computing device cluster 110, and generates computing power anomaly information when computing power anomalies are detected, and sends the computing power anomaly information to the training server through a preset interface; the training server is used to execute the training task of the deep learning model, thereby realizing the training of the deep learning model on the heterogeneous computing device cluster 110. In the embodiment of the present application, the training server can also interrupt the training of the deep learning model on the heterogeneous computing device cluster 110 in response to the computing power anomaly information sent by the computing power detection server, and automatically continue the training of the deep learning model after the interruption.

[0079] It should be noted that the server involved in the embodiments of the present application can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network) and big data and artificial intelligence platforms.

[0080] In an exemplary embodiment, the heterogeneous computing device cluster 110 and the server 120 can both be node devices in the blockchain system, and can share the acquired and generated information with other node devices in the blockchain system to achieve information sharing between multiple node devices. Multiple node devices in the blockchain system can be configured with the same blockchain, which is composed of multiple blocks, and the adjacent blocks have an association relationship, so that when the data in any block is tampered, it can be detected by the next block, thereby preventing the data in the blockchain from being tampered with, and ensuring the security and reliability of the data in the blockchain.

[0081] The embodiments of the present application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, etc.

[0082] Among them, cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the cloud computing business model application, which can form a resource pool and be used on demand, flexibly and conveniently. With the rapid development and application of the Internet industry, in the future, each item may have its own identification mark, and all need to be transmitted to the background system for logical processing. Data of different levels will be processed separately, and all kinds of industry data need strong system backing support, which can be achieved through cloud computing.

[0083] Cloud computing is a computing model that distributes computing tasks on a resource pool composed of a large number of computers, allowing various application systems to obtain computing power, storage space and information services as needed. The network that provides resources is called a "cloud". From the user's perspective, the resources in the "cloud" are infinitely scalable and can be obtained at any time, used on demand, expanded at any time, and paid for by use.

[0084] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level technology and software-level technology. The basic technologies of artificial intelligence generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, and mechatronics. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0085] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0086] See also Figure 2 , which is a flow chart of an exception handling method provided in an embodiment of the present application, which can be applied to Figure 1 Server in. It should be noted that this specification provides method operation steps as described in the embodiments or flow charts, but more or fewer operation steps may be included based on routine or non-creative work. The order of steps listed in the embodiments is only one way of executing the steps among many orders, and does not represent the only order of execution. When the actual system or product is executed, it can be executed in sequence or in parallel (for example, in a parallel processor or multi-threaded processing environment) according to the method shown in the embodiments or drawings. Specific examples Figure 2 As shown, the method may include:

[0087] S201, in the process of training a deep learning model based on a heterogeneous computing device cluster, detecting abnormal computing power information of the heterogeneous computing device cluster.

[0088] Among them, the computing power anomaly information represents an abnormal heterogeneous computing device with abnormal computing power indicators, and the abnormal computing power indicators corresponding to the abnormal heterogeneous computing device.

[0089] Computing power refers to the ability of heterogeneous computing devices to perform model training calculations. The computing power indicator is used to characterize the computing power of heterogeneous computing devices.

[0090] In some exemplary embodiments, the computing power of heterogeneous computing devices can be measured from a functional perspective, and the computing power index can include at least one functional index, each functional index represents a preset operating state of the hardware card, and the preset operating state is an operating state indicating that the hardware card is abnormal. Figure 3 As shown, when the above step S201 is implemented, for each heterogeneous computing device in the heterogeneous computing device cluster, the following steps may be performed:

[0091] S301, performing detection on at least one preset operating state of a hardware card in the heterogeneous computing device to obtain a state detection result of the heterogeneous computing device corresponding to each preset operating state.

[0092] The status detection result indicates whether the hardware card is in a corresponding preset operating state.

[0093] Among them, at least one preset operating state can be set and adjusted based on the abnormal situation of the hardware card in actual operation.

[0094] In a specific implementation, at least one preset operating state is detected for the hardware card in the heterogeneous computing device. If a preset operating state is detected, a first state detection result is generated corresponding to the preset operating state, and the first state detection result indicates that the hardware card is in the preset operating state. Conversely, if a preset operating state is not detected, a second state detection result is generated corresponding to the preset operating state, and the second state detection result indicates that the hardware card is not in the preset operating state.

[0095] S303: If the state detection results corresponding to each preset running state of the heterogeneous computing device include a first state detection result, it is determined that the heterogeneous computing device has an abnormality in the target function indicator.

[0096] Among them, the first state detection result represents that the hardware card has a preset operating state corresponding to the first state detection result, referred to as the first preset operating state, and the target function indicator is the function indicator corresponding to the first preset operating state, that is, the abnormal function indicator.

[0097] Specifically, if there is a first state detection result in at least one state detection result corresponding to the heterogeneous computing device, it indicates that a corresponding preset operating state of the hardware card is detected. Then, it can be determined that the heterogeneous computing device has an abnormality in the functional indicator corresponding to the corresponding preset operating state, and thus the heterogeneous computing device is an abnormal heterogeneous computing device. At this time, the following step S305 can be executed.

[0098] It is understandable that an abnormal heterogeneous computing device may have one or more abnormal target function indicators.

[0099] It can be understood that if the state detection results corresponding to each preset operating state of the heterogeneous computing device do not contain the first state detection result, it can be determined that the heterogeneous computing device does not have abnormal functional indicators, and the heterogeneous computing device can be considered to be a normal heterogeneous computing device.

[0100] S305: Generate computing power anomaly information of the heterogeneous computing device cluster based on the address information of the heterogeneous computing device and the above-mentioned target function indicator.

[0101] It can be understood that the number of abnormal heterogeneous computing devices detected through steps S301 to S303 may be zero, one, or more. Then, when at least one abnormal heterogeneous computing device is detected, the computing power abnormality information of the heterogeneous computing device cluster may be generated based on the address information of each abnormal heterogeneous computing device and the target function indicator corresponding to the abnormal heterogeneous computing device. The address information of the heterogeneous computing device may be IP information.

[0102] To facilitate understanding of the above implementation, taking the hardware card as a GPU card as an example, the preset running state of each GPU computing device in the GPU computing device cluster may be detected according to the following Table 1:

[0103] Table 1

[0104]

[0105] If a GPU device is detected to have abnormal function indicators as shown in Table 1 above, the computing power abnormality information can be generated based on the IP address of the GPU device and the corresponding abnormal function indicators.

[0106] In actual applications, the above steps S301 to S305 may be performed according to a preset period. The preset period may be in seconds or minutes. The length of the specific preset period may be set according to actual needs.

[0107] It is understandable that the computing power anomaly information may also include other information according to actual needs, such as detection time information and description information of abnormal function indicators, etc.

[0108] In the above implementation, by detecting at least one preset operating state of the hardware card in each heterogeneous computing device, and when there is a first state detection result representing that the hardware card has the corresponding preset operating state in the detection results corresponding to each preset operating state of the heterogeneous computing device, it is determined that the heterogeneous computing device has an abnormality in the target functional indicator, i.e., the functional indicator corresponding to the corresponding preset operating state, and then the computing power abnormality information of the heterogeneous computing device cluster can be quickly and accurately generated based on the address information of the heterogeneous computing device and the target functional indicator.

[0109] In some exemplary embodiments, the computing power of heterogeneous computing devices can also be measured from the perspective of performance, and the computing power index can include at least one performance index. Figure 4 As shown, when the above step S201 is implemented, for each heterogeneous computing device in the heterogeneous computing device cluster, the following steps may also be performed:

[0110] S401, obtaining performance indicator data corresponding to each performance indicator of the heterogeneous computing device.

[0111] Among them, at least one performance indicator can be set and adjusted based on the needs in actual applications. For example, at least one performance indicator can include GPU utilization, video memory utilization, network bandwidth throughput, and disk IO throughput.

[0112] S403: Determine the fluctuation range of each performance indicator of the heterogeneous computing device according to the performance indicator data of each performance indicator of the heterogeneous computing device.

[0113] Specifically, the fluctuation range of each performance indicator of the heterogeneous computing device can be obtained based on the performance indicator data obtained at the current time and the performance indicator data obtained at the last time. In a specific implementation, the difference between the performance indicator data of the performance indicator at the current time and the performance indicator data at the last time can be calculated, and then the ratio between the absolute value of the difference and the performance indicator data at the last time can be calculated, so as to obtain the fluctuation range of the performance indicator of the heterogeneous computing device corresponding to the performance indicator.

[0114] S405, comparing the fluctuation range of each performance indicator of the heterogeneous computing device with a preset fluctuation range threshold corresponding to each performance indicator, to obtain a performance detection result of each performance indicator of the heterogeneous computing device.

[0115] The performance test result indicates whether the fluctuation range of the corresponding performance indicator exceeds the preset fluctuation range threshold corresponding to the performance indicator.

[0116] The preset fluctuation amplitude threshold corresponding to each performance indicator can be set based on the needs in actual applications, and can be set to be the same or different.

[0117] Taking at least one performance indicator including GPU utilization, video memory utilization, network bandwidth throughput and disk IO throughput as an example, since video memory and disk IO belong to the load of stable resources during model training, video memory utilization and disk IO throughput can be set with a lower preset fluctuation amplitude threshold, such as 10%; and for GPU utilization, since the use of GPU will fluctuate significantly with the change of data volume, a looser preset fluctuation amplitude threshold can be set for GPU utilization, such as 20%; and for network bandwidth throughput, since the network bandwidth is not a smooth curve during large model training, and network traffic does not exist in real time, the preset fluctuation amplitude threshold of network bandwidth throughput is mainly used to indicate a sudden drop in network traffic, and can be set to 50%.

[0118] In a specific implementation, the fluctuation amplitude of each performance indicator of the heterogeneous computing device can be compared with the preset fluctuation amplitude threshold corresponding to each performance indicator. If the fluctuation amplitude of the performance indicator exceeds the preset fluctuation amplitude threshold of the performance indicator, a first performance detection result is generated for the performance indicator, and the first performance detection result indicates that the fluctuation amplitude of the corresponding performance indicator exceeds its preset fluctuation amplitude threshold. Conversely, if the fluctuation amplitude of the performance indicator does not exceed the preset fluctuation amplitude threshold of the performance indicator, a second performance detection result is generated for the performance indicator, and the second performance detection result indicates that the fluctuation amplitude of the corresponding performance indicator does not exceed its preset fluctuation amplitude threshold.

[0119] S407: If the first performance detection result exists in the performance detection results corresponding to the performance indicators of the heterogeneous computing device, it is determined that the target performance indicator of the heterogeneous computing device is abnormal.

[0120] Among them, the target performance indicator is a performance indicator corresponding to the above-mentioned first performance detection result, and the first performance detection result represents that the fluctuation range of the target performance indicator exceeds the preset fluctuation range threshold value corresponding to the target performance indicator.

[0121] Specifically, if there is a first performance test result in the performance test results corresponding to each performance indicator of the heterogeneous computing device, it indicates that the performance indicator corresponding to the first performance test result, that is, the target performance indicator, is abnormal, and thus the heterogeneous computing device is an abnormal heterogeneous computing device. At this time, the following step S409 can be executed.

[0122] It is understandable that an abnormal heterogeneous computing device may have one or more abnormal target performance indicators.

[0123] It can be understood that if the first performance test result does not exist in the performance test results corresponding to each performance indicator of the heterogeneous computing device, it can be determined that the heterogeneous computing device does not have an abnormal performance indicator, and the heterogeneous computing device can be considered to be a normal heterogeneous computing device.

[0124] S409: Generate computing power anomaly information of the heterogeneous computing device cluster based on the address information of the heterogeneous computing device and the above-mentioned target performance indicator.

[0125] It can be understood that the number of abnormal heterogeneous computing devices detected through steps S401 to S407 may be zero, one, or more. Then, when at least one abnormal heterogeneous computing device is detected, the computing power abnormality information of the heterogeneous computing device cluster may be generated based on the address information of each abnormal heterogeneous computing device and the target performance indicator corresponding to the abnormal heterogeneous computing device. The address information of the heterogeneous computing device may be IP information.

[0126] In actual applications, the above steps S401 to S409 may be performed according to a preset period, which may be in seconds or minutes. The length of the specific preset period may be set according to actual needs.

[0127] It can be understood that the aforementioned steps S301 to S305 and steps S401 to S407 can be executed synchronously, so when generating the computing power abnormality information of the heterogeneous computing device cluster, the computing power abnormality information can be generated based on the abnormal function indicators and / or abnormal performance indicators of the abnormal heterogeneous computing devices.

[0128] In the above implementation, by measuring the computing power of heterogeneous computing devices from both the performance perspective and the functional perspective, abnormal heterogeneous computing devices in a heterogeneous computing device cluster can be detected more quickly and accurately, which is beneficial to improving the detection efficiency and accuracy of computing power anomaly information of a heterogeneous computing device cluster, and further beneficial to improving the utilization rate of computing power resources during deep learning model training.

[0129] S203: If the above-mentioned computing power abnormality information is detected, the training of the deep learning model on the heterogeneous computing device cluster is interrupted.

[0130] Specifically, if abnormal computing power information of a heterogeneous computing device cluster is detected, it indicates that a computing power abnormality has occurred in the heterogeneous computing device cluster. At this time, the training of the deep learning model on the heterogeneous computing device cluster can be interrupted.

[0131] In actual applications, the training server implements the training of the deep learning model on the heterogeneous computing device cluster by executing the training task of the deep learning model, wherein the training task can be set based on the actual application requirements of the deep learning model after training, for example, the training task can be an image recognition task, a speech recognition task, etc. In other words, during the training of the deep learning model, if the training task process of the corresponding training task is running on the training server, the training of the deep learning model on the heterogeneous computing device cluster can be interrupted by ending the training task process.

[0132] S205: Search the current task processing strategy set for a target task processing strategy that matches the abnormal computing power indicator.

[0133] Among them, the task processing strategies in the current task processing strategy set correspond to different computing power indicators, and each task processing strategy represents the training resources that need to be loaded when the corresponding computing power indicator is abnormal, and the training resources include the training status information corresponding to the deep learning model when the training is interrupted. The current task processing strategy set can be updated during the training process of the deep learning model.

[0134] Among them, the training state information can be used to initialize the training state of the model. Exemplarily, the training state information can include but is not limited to model weight parameters, EPOCH, optimizer state information, scheduler state information, etc.

[0135] It should be noted that the training resources represented by the task processing strategies corresponding to different computing power indicators match the computing power indicators, so that when the computing power indicators are abnormal, the training of the deep learning model can be restarted and continued by loading the corresponding training resources. For example, for the computing power indicator "network card status", if the computing power indicator is abnormal, it is possible that the training data required by each GPU device in the cluster has not been fully transmitted. In this case, the training resources that need to be loaded represented by the task processing strategy corresponding to the computing power indicator should include the aforementioned training status information and training data; and for the computing power indicator "GPU card ECC", since the abnormality of this computing power indicator usually does not affect the transmission of training data during training, the training resources that need to be loaded represented by the task processing strategy corresponding to the computing power indicator may not include training data.

[0136] Table 2 below is an example of the current task processing strategy set, which is stored in the form of a data table. Among them, task processing strategy 1 indicates that when the computing power indicator "GPU card ECC" is abnormal, the training resources that need to be loaded are training status information; task processing strategy 2 indicates that when the computing power indicator "GPU card Xid" is abnormal, the training resources that need to be loaded are training status information; task processing strategy 3 indicates that when the computing power indicator "GPU card temperature" is abnormal, the training resources that need to be loaded are training status information and training data.

[0137] Table 2

[0138] Task processing strategy Hashrate Indicators Training status information Training data loading Memory cache loading Task processing strategy 1 GPU Card ECC yes no no Task processing strategy 2 GPU card Xid yes no no Task processing strategy 3 GPU card temperature yes yes no

[0139] It can be understood that the above Table 2 only gives examples of several types of training resources (training status information, training data, memory cache). In actual applications, the types of training resources can be expanded based on actual needs on the basis of what is shown in the above Table 2.

[0140] In a specific implementation, after interrupting the training of the deep learning model on a heterogeneous computing device cluster, if a target task processing strategy matching the abnormal computing power indicator is found from the current task processing strategy set, the following step S207 can be executed.

[0141] Taking the current task processing strategy set shown in Table 2 above as an example, if the computing power abnormality information detected in step S201 indicates that the GPU card temperature (i.e., the abnormal computing power indicator) of the heterogeneous computing device 1 is abnormal, then when executing step S205 above, the task processing strategy matching the "GPU card temperature" can be found from Table 2 above. Obviously, it can be found that the target task processing strategy is task processing strategy 3, so the following step S207 can be executed.

[0142] S207: If the target task processing strategy is found, restart the training of the deep learning model on the target heterogeneous computing device cluster based on the target task processing strategy.

[0143] The target heterogeneous computing device cluster includes heterogeneous computing devices except abnormal heterogeneous computing devices.

[0144] In the specific implementation, the training task of the deep learning model can be executed based on the target task processing strategy, and then the training of the deep learning model on the heterogeneous computing device cluster can be restarted to realize the automatic continuation of the deep learning model.

[0145] In an exemplary embodiment, Figure 5 As shown, in the aforementioned step S205, if the target task processing strategy matching the abnormal computing power indicator is not found from the current task processing strategy set, the following steps S209 to S213 may be executed:

[0146] S209: If the target task processing strategy is not found, the task processing configuration information input for the abnormal computing power indicator is obtained based on the task processing configuration interface.

[0147] Among them, the task processing configuration information represents the training resources that need to be loaded when abnormal computing power indicators appear.

[0148] Specifically, if the target task processing strategy that matches the abnormal computing power indicator is not found in the current task processing strategy set, it indicates that the corresponding computing power indicator is abnormal for the first time. Considering that the training resources that need to be loaded during the restart process corresponding to the abnormality of each computing power indicator may be different, for example, the abnormality of the network card status corresponds to the restart process. Not only does it need to load the training status information, but it also needs to load the training data in an overwriting manner; and in the restart process corresponding to the abnormality of the GPU card ECC, after loading the training status information, the data transmission can maintain the original configuration, and there is no need to overwrite the training data. Therefore, in order to ensure the accuracy of the deep learning model's continued training after an interruption, it is necessary to configure the task processing strategy for the computing power indicator that has the first abnormality.

[0149] When configuring the task processing strategy, the task processing configuration interface can be displayed based on the terminal of the task processing node. The task processing configuration interface can include an abnormal computing power indicator input area and a training resource input area, wherein the abnormal computing power indicator input area is used to input abnormal computing power indicators, and the training resource input area is used to input training resources that need to be loaded when the computing power indicators entered in the abnormal computing power indicator input area are abnormal, and the training resources include the training status information of the model when the training is interrupted. When confirming that the configuration is complete, the task processing configuration information for the abnormal computing power indicator can be obtained based on the input content of the abnormal computing power indicator input area and the training resource input area in the task processing configuration interface.

[0150] In actual applications, in order to improve the efficiency of task processing strategy configuration, the training resource input area may include multiple preset resource options. When input operations are performed in the training resource input area, the multiple preset resource options may be displayed for selection, and then the selected preset resource may be determined as the training resource to be loaded in response to the selection operation for the multiple preset resource options, and the selected preset resource may be used as the input content of the training resource input area. Exemplarily, the multiple preset resource options may include checkpoints, training data, memory cache, etc., which may be expanded and adjusted based on the needs in actual applications.

[0151] S211, based on the above task processing configuration information, restart the training of the deep learning model on the target heterogeneous computing device cluster.

[0152] Specifically, for the first abnormal computing power indicator (i.e., abnormal computing power indicator), after obtaining the task processing configuration information configured for the abnormal computing power indicator, the training task of the deep learning model can be executed based on the task processing configuration information, and then the training of the deep learning model on the target heterogeneous computing device cluster can be restarted to achieve semi-automatic continued training of the deep learning model.

[0153] S213: Using the task processing configuration information as the task processing strategy in the current task processing strategy set, the current task processing strategy set is updated.

[0154] Specifically, the updated current task processing strategy set includes the above-mentioned task processing configuration information for the abnormal computing power indicator, and then when the abnormal computing power indicator appears again, the corresponding target task processing strategy can be found from the current task processing strategy set, and the above-mentioned steps S201 to S207 can be executed to realize the automatic continued training of the deep learning model.

[0155] In the above implementation, through steps S209 to S211, for the computing power indicator that appears abnormal for the first time, the training of the deep learning model on the cluster is restarted in a semi-automatic mode after the training is interrupted, and the task processing configuration information is updated as the task processing strategy to the current task processing strategy set, so as to realize the breakpoint resumption training from the semi-automatic mode to the automatic mode, which is beneficial to shorten the deep learning model training cycle and improve the utilization rate of heterogeneous computing devices; and the current task processing strategy set will be continuously iterated and improved, that is, the problem of computing power of heterogeneous computing devices can be continuously converged and iterated into the automated processing flow of training, so that the abnormal resumption of training in the deep learning model training can support a fully automated mode, which is beneficial to reduce the idle waiting situation of cluster computing power due to faults, especially the computing power of large models with tens of thousands of cards, which can significantly reduce the cost of model training.

[0156] In an exemplary embodiment, in order to restart the training of the deep learning model more accurately and quickly, after the aforementioned step S203 interrupts the training of the deep learning model on the heterogeneous computing device cluster, the method may further include: marking abnormal heterogeneous computing devices in the heterogeneous computing device cluster based on computing power abnormality information.

[0157] Then, before restarting the training of the deep learning model on the target heterogeneous computing device cluster, the method may further include: determining a cluster composed of unmarked heterogeneous computing devices in the heterogeneous computing device cluster as the target heterogeneous computing device cluster.

[0158] In a specific implementation, since the computing power anomaly information represents an abnormal heterogeneous computing device with abnormal computing power indicators, as well as the abnormal computing power indicators corresponding to the abnormal heterogeneous computing device, the abnormal heterogeneous computing device can be determined by parsing the computing power anomaly information. For example, the IP address information of the abnormal heterogeneous computing device can be obtained, and then the abnormal heterogeneous computing device in the heterogeneous computing device cluster can be marked. For example, the abnormal heterogeneous computing device can be marked with a preset tag, which indicates that the heterogeneous computing device is abnormal.

[0159] Therefore, before restarting the training of the deep learning model, the heterogeneous computing devices in the heterogeneous computing device cluster that do not have the preset tag can be used as heterogeneous computing devices in the target heterogeneous computing device cluster, so that the training of the deep learning model on the target heterogeneous computing device cluster can be restarted based on the target task processing strategy.

[0160] In an exemplary embodiment, in order to accurately determine the training status information of the deep learning model when the training is interrupted when restarting the training, the exception handling method of the embodiment of the present application may also include: during the training process of the deep learning model, recording the training status of the deep learning model at preset time intervals to obtain the training status information corresponding to each time point.

[0161] Then, when executing the target task processing strategy to restart the training of the deep learning model on the target heterogeneous computing device cluster, step S207 may include: determining the target time point that is closest to the interrupted training among each time point, and using the training status information corresponding to the target time point as the training status information corresponding to the training resources in the target task processing strategy; and then restarting the training of the deep learning model on the target heterogeneous computing device cluster based on the target task processing strategy.

[0162] Specifically, the preset time interval can be set based on actual needs, for example, it can be recorded once per second, so that the training status information at the second level can be obtained. In practical applications, the training status information of the deep learning model at each time point can be recorded based on the checkpoint mechanism, and the checkpoint file at each time point can be obtained. Among them, the checkpoint mechanism is a mechanism for periodically saving the training status of the model. By periodically saving the parameter status of the model, the model status can be restored from the checkpoint when there is a problem in the training.

[0163] Then, when restarting the training of the deep learning model on the target heterogeneous computing device cluster based on the target task processing strategy, the checkpoint file at the target time point and other training resources indicated by the target task processing strategy can be loaded.

[0164] In order to facilitate understanding of the technical solution of the embodiment of the present application, Figure 6 The following example illustrates the architecture of

[0165] like Figure 6 The architecture shown is divided into two closed-loop processes: a semi-automated closed-loop process on the first layer and a fully automated closed-loop process on the second layer. The sources of computing power anomaly information based on the two layers of processes are both indicator data of functions and performance detected by the GPU computing power detection side. Among them, the semi-automated closed-loop process on the first layer is applied when the anomaly of the computing power indicator is first discovered. After the computing power detection side sends the computing power anomaly information to the first layer in the form of an alarm, the original training task of the deep learning model is first interrupted. In the process of restarting the deep learning model training, the first layer pulls up a new training task based on the manually input re-initialized operation items (that is, the task processing configuration information for the abnormal computing power indicators) to restart the training of the deep learning model on the GPU device cluster after the abnormal GPU device is removed, and it involves the restart strategy (that is, Figure 6 The task processing configuration information in the system) can be passed to the second-layer closed-loop process in the form of "abnormal case feedback", and then integrated into the second-layer automation logic, so that when the same computing power indicator is detected as abnormal again by the computing power detection side, the second-layer closed-loop process can automatically restart the model training. After the computing power detection side issues computing power abnormality information in the form of an alarm, the second layer first interrupts the original training task of the deep learning model, and then automatically starts a new training task based on the task processing logic corresponding to the computing power indicator that has experienced an abnormality, so as to automatically restart the training of the deep learning model on the GPU device cluster after the abnormal GPU device is removed, so as to achieve rapid recovery and training without manual intervention.

[0166] In actual applications, considering that the first layer needs to configure task processing for abnormal computing power indicators to obtain corresponding task processing configuration information, in order to improve the accuracy of task processing configuration, the computing power detection side can also generate a performance trend chart of each GPU device during the training process based on the performance data detected during the training process and send it to the first-layer closed-loop process, so that relevant personnel in the first-layer closed-loop process can analyze the performance trend chart to determine the training resources that need to be configured, thereby improving the accuracy of the corresponding task processing configuration information.

[0167] The above implementation method realizes the transition of breakpoint resume training from semi-automatic mode to automatic mode through double-layer closed-loop processing, which is beneficial to shorten the training cycle of large models and improve the utilization rate of GPU equipment; and through the constructed double-layer closed-loop processing, the GPU computing power problem is continuously converged and iterated into the automated processing flow, which is beneficial to reduce the idle waiting of GPU computing power caused by faults, especially for large models with computing power of tens of thousands of cards, which significantly saves training costs; in addition, double-layer closed-loop processing is beneficial to the convergence of fault problems and the resolution of fault problems by operation and maintenance personnel, especially the continuous second-layer automated processing flow, which completely decouples the training of the model and the operation and maintenance response of computing power, alleviating the pressure on computing power resource operation and maintenance personnel.

[0168] like Figure 7 As shown, the embodiment of the present application also provides another exception handling method, which includes:

[0169] S701, in the process of performing language processing training on a natural language processing model based on a heterogeneous computing device cluster, detecting abnormal computing power information of the heterogeneous computing device cluster.

[0170] Among them, the computing power anomaly information represents an abnormal heterogeneous computing device with abnormal computing power indicators, and the abnormal computing power indicators corresponding to the abnormal heterogeneous computing device.

[0171] S703: If the above-mentioned computing power abnormality information is detected, the language processing training of the natural language processing model on the above-mentioned heterogeneous computing device cluster is interrupted.

[0172] S705: Search the current task processing strategy set for a target task processing strategy that matches the abnormal computing power indicator.

[0173] Among them, the task processing strategies in the current task processing strategy set correspond to different computing power indicators, and each task processing strategy represents the training resources that need to be loaded when the corresponding computing power indicator is abnormal. The training resources include the training status information corresponding to the natural language processing model when the language processing training is interrupted.

[0174] S707: If the target task processing strategy is found, restart the language processing training of the natural language processing model on the target heterogeneous computing device cluster based on the target task processing strategy.

[0175] The target heterogeneous computing device cluster includes heterogeneous computing devices other than the abnormal heterogeneous computing device.

[0176] S709: If the target task processing strategy is not found, the task processing configuration information input for the abnormal computing power indicator is obtained based on the task processing configuration interface.

[0177] Among them, the task processing configuration information represents the training resources that need to be loaded when the abnormal computing power indicators appear.

[0178] S711: Restart the language processing training of the natural language processing model on the target heterogeneous computing device cluster based on the task processing configuration information.

[0179] S713: Use the task processing configuration information as a task processing strategy in the current task processing strategy set to update the current task processing strategy set.

[0180] In a specific implementation, the detailed contents of the above steps S701 to S713 can refer to the relevant descriptions of the above steps S201 to S213 in the embodiment of the present application, and will not be repeated here.

[0181] In some exemplary embodiments, in order to restart the language processing training of the natural language processing model more accurately and quickly, after the aforementioned step S703 interrupts the language processing training of the natural language processing model on the heterogeneous computing device cluster, the method may further include: marking abnormal heterogeneous computing devices in the heterogeneous computing device cluster based on computing power abnormality information.

[0182] Then, before restarting the language processing training of the natural language processing model on the target heterogeneous computing device cluster, the method may further include: determining a cluster composed of unlabeled heterogeneous computing devices in the heterogeneous computing device cluster as the target heterogeneous computing device cluster.

[0183] In a specific implementation, since the computing power anomaly information represents an abnormal heterogeneous computing device with abnormal computing power indicators, as well as the abnormal computing power indicators corresponding to the abnormal heterogeneous computing device, the abnormal heterogeneous computing device can be determined by parsing the computing power anomaly information. For example, the IP address information of the abnormal heterogeneous computing device can be obtained, and then the abnormal heterogeneous computing device in the heterogeneous computing device cluster can be marked. For example, the abnormal heterogeneous computing device can be marked with a preset tag, which indicates that the heterogeneous computing device is abnormal.

[0184] Therefore, before restarting the language processing training of the natural language processing model, the heterogeneous computing devices in the heterogeneous computing device cluster that do not have the preset tag can be used as heterogeneous computing devices in the target heterogeneous computing device cluster, so that the language processing training of the natural language processing model on the target heterogeneous computing device cluster can be restarted based on the target task processing strategy.

[0185] In an exemplary embodiment, in order to accurately determine the training status information of the natural language processing model when the language processing training is interrupted when restarting the language processing training, the exception handling method of the embodiment of the present application may also include: during the language processing training process of the natural language processing model, recording the training status of the natural language processing model at preset time intervals to obtain the training status information corresponding to each time point.

[0186] Then, when executing the target task processing strategy to restart the language processing training of the natural language processing model on the target heterogeneous computing device cluster, step S707 may include: determining the target time point closest to the interrupted language processing training among each time point, and using the training status information corresponding to the target time point as the training status information corresponding to the training resources in the target task processing strategy; and then restarting the language processing training of the natural language processing model on the target heterogeneous computing device cluster based on the target task processing strategy.

[0187] Specifically, the preset time interval can be set based on actual needs, for example, it can be recorded once per second, so that the training status information at the second level can be obtained. In practical applications, the training status information of the natural language processing model at each time point can be recorded based on the checkpoint mechanism, and the checkpoint file at each time point can be obtained. Among them, the checkpoint mechanism is a mechanism for periodically saving the training status of the model. By periodically saving the parameter status of the model, the model status can be restored from the checkpoint when a problem occurs in the training.

[0188] Then, when restarting the language processing training of the natural language processing model on the target heterogeneous computing device cluster based on the target task processing strategy, the checkpoint file at the target time point and other training resources indicated by the target task processing strategy can be loaded.

[0189] Corresponding to the exception handling methods provided in the above-mentioned embodiments, the embodiments of the present application also provide an exception handling device. Since the exception handling device provided in the embodiments of the present application corresponds to the exception handling methods provided in the above-mentioned embodiments, the implementation methods of the aforementioned exception handling methods are also applicable to the exception handling device provided in this embodiment and will not be described in detail in this embodiment.

[0190] See also Figure 8 , which is a schematic diagram of the structure of an exception handling device provided in an embodiment of the present application, the device has the function of implementing the exception handling method in the above method embodiment, and the function can be implemented by hardware or by hardware executing corresponding software. Figure 8 As shown, the exception handling device 800 may include:

[0191] The first computing power anomaly detection module 810 is used to detect computing power anomaly information of the heterogeneous computing device cluster during the process of training the deep learning model based on the heterogeneous computing device cluster; the computing power anomaly information represents an abnormal heterogeneous computing device with abnormal computing power index, and an abnormal computing power index corresponding to the abnormal heterogeneous computing device;

[0192] A first training interruption module 820 is used to interrupt the training of the deep learning model on the heterogeneous computing device cluster when the computing power abnormality information is detected;

[0193] The first processing strategy search module 830 is used to search for a target task processing strategy that matches the abnormal computing power indicator from the current task processing strategy set; the task processing strategies in the current task processing strategy set correspond to different computing power indicators, and each task processing strategy represents a training resource that needs to be loaded when the corresponding computing power indicator is abnormal, and the training resource includes the training state information corresponding to the deep learning model when the training is interrupted;

[0194] The first training restart module 840 is used to restart the training of the deep learning model on the target heterogeneous computing device cluster based on the target task processing strategy when the target task processing strategy is found; the target heterogeneous computing device cluster includes heterogeneous computing devices other than the abnormal heterogeneous computing device.

[0195] In some exemplary embodiments, the apparatus 800 may further include:

[0196] A first task processing configuration module is used to obtain task processing configuration information input for the abnormal computing power indicator based on the task processing configuration interface when the target task processing strategy is not found; the task processing configuration information represents the training resources that need to be loaded when the abnormal computing power indicator appears;

[0197] A third training restart module is used to restart the training of the deep learning model on the target heterogeneous computing device cluster based on the task processing configuration information;

[0198] The first task processing strategy set updating module is used to use the task processing configuration information as the task processing strategy in the current task processing strategy set to update the current task processing strategy set.

[0199] In some exemplary embodiments, the computing power indicator includes at least one functional indicator, each of which represents a preset operating state of the hardware card; the first computing power anomaly detection module 810 includes:

[0200] A first detection module is used to detect at least one of the preset operating states of a hardware card in each heterogeneous computing device in the heterogeneous computing device cluster, and obtain a state detection result of the heterogeneous computing device corresponding to each of the preset operating states; the state detection result indicates whether the hardware card has a corresponding preset operating state;

[0201] A first determination module is configured to determine that the heterogeneous computing device has an abnormality in a target functional indicator when a first state detection result exists in the state detection results corresponding to each of the preset operating states of the heterogeneous computing device; the first state detection result indicates that the hardware card has a first preset operating state corresponding to the first state detection result, and the target functional indicator is a functional indicator corresponding to the first preset operating state;

[0202] The first generating module is used to generate computing power anomaly information of the heterogeneous computing device cluster based on the address information of the heterogeneous computing device and the target function indicator.

[0203] In some exemplary embodiments, the computing power indicator includes at least one performance indicator; the first computing power anomaly detection module 810 includes:

[0204] A data acquisition module, used for acquiring, for each heterogeneous computing device in the heterogeneous computing device cluster, performance indicator data corresponding to each performance indicator of the heterogeneous computing device;

[0205] A second determination module is used to determine the fluctuation range of each performance indicator corresponding to the heterogeneous computing device according to the performance indicator data corresponding to each performance indicator of the heterogeneous computing device;

[0206] The second detection module is used to compare the fluctuation range of each performance indicator of the heterogeneous computing device with the preset fluctuation range threshold corresponding to each performance indicator, and obtain the performance detection result of each performance indicator of the heterogeneous computing device; the performance detection result indicates whether the fluctuation range of the corresponding performance indicator exceeds the preset fluctuation range threshold corresponding to the performance indicator;

[0207] A third determination module is used to determine that there is an abnormality in a target performance indicator of the heterogeneous computing device when there is a first performance detection result in the performance detection results corresponding to each performance indicator of the heterogeneous computing device; the target performance indicator is a performance indicator corresponding to the first performance detection result, and the first performance detection result indicates that the fluctuation range of the target performance indicator exceeds a preset fluctuation range threshold corresponding to the target performance indicator;

[0208] The second generating module is used to generate computing power anomaly information of the heterogeneous computing device cluster based on the address information of the heterogeneous computing device and the target performance indicator.

[0209] In some exemplary embodiments, the device 80 may further include:

[0210] An abnormal marking module, used for marking the abnormal heterogeneous computing device in the heterogeneous computing device cluster based on the computing power abnormality information;

[0211] The fourth determination module is used to determine a cluster consisting of unmarked heterogeneous computing devices in the heterogeneous computing device cluster as a target heterogeneous computing device cluster.

[0212] In some exemplary embodiments, the apparatus 800 may further include:

[0213] A first training state recording module is used to record the training state of the deep learning model at preset time intervals during the training process of the deep learning model to obtain training state information corresponding to each time point;

[0214] The first training restart module 840 includes:

[0215] A fifth determination module is used to determine a target time point that is closest to interrupting the training among the time points, and use the training state information corresponding to the target time point as the training state information corresponding to the training resource in the target task processing strategy;

[0216] The first training restart submodule is used to restart the training of the deep learning model on the target heterogeneous computing device cluster based on the target task processing strategy.

[0217] See also Fig. 9 , which is a schematic diagram of the structure of an exception handling device provided in an embodiment of the present application, such as Fig. 9 As shown, the exception handling device 900 may include:

[0218] The second computing power anomaly detection module 910 is used to detect computing power anomaly information of the heterogeneous computing device cluster during the process of language processing training of the natural language processing model based on the heterogeneous computing device cluster; the computing power anomaly information represents an abnormal heterogeneous computing device with abnormal computing power indicators, and an abnormal computing power indicator corresponding to the abnormal heterogeneous computing device;

[0219] A second training interruption module 920 is used to interrupt the language processing training of the natural language processing model on the heterogeneous computing device cluster when the computing power abnormality information is detected;

[0220] The second processing strategy search module 930 is used to search for a target task processing strategy that matches the abnormal computing power indicator from the current task processing strategy set; the task processing strategies in the current task processing strategy set correspond to different computing power indicators, and each task processing strategy represents the training resources that need to be loaded when the corresponding computing power indicator is abnormal; the training resources include the training state information corresponding to the natural language processing model when the language processing training is interrupted;

[0221] The second training restart module 940 is used to restart the language processing training of the natural language processing model on the target heterogeneous computing device cluster based on the target task processing strategy when the target task processing strategy is found; the target heterogeneous computing device cluster includes heterogeneous computing devices except the abnormal heterogeneous computing device.

[0222] In some exemplary embodiments, the apparatus 900 further includes:

[0223] A second task processing configuration module is used to obtain task processing configuration information input for the abnormal computing power indicator based on the task processing configuration interface when the target task processing strategy is not found; the task processing configuration information represents the training resources that need to be loaded when the abnormal computing power indicator appears;

[0224] a fourth training restart module, configured to restart the language processing training of the natural language processing model on the target heterogeneous computing device cluster based on the task processing configuration information;

[0225] The second task processing strategy set updating module is used to use the task processing configuration information as the task processing strategy in the current task processing strategy set to update the current task processing strategy set.

[0226] In some exemplary embodiments, the apparatus 900 further includes:

[0227] A second training state recording module is used to record the training state of the natural language processing model at preset time intervals during the language processing training process of the natural language processing model to obtain training state information corresponding to each time point;

[0228] The second training restart module 940 includes:

[0229] A sixth determination module is used to determine a target time point that is closest to interrupting the language processing training among the time points, and use the training state information corresponding to the target time point as the training state information corresponding to the training resource in the target task processing strategy;

[0230] The second training restart submodule is used to restart the language processing training of the natural language processing model on the target heterogeneous computing device cluster based on the target task processing strategy.

[0231] It should be noted that the device provided in the above embodiment, when implementing its functions, is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0232] An embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement any one of the exception handling methods provided in the above method embodiments.

[0233] The memory can be used to store software programs and modules. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, application programs required for functions, etc.; the data storage area can store data created according to the use of the device, etc. In addition, the memory can include a high-speed random access memory and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory can also include a memory controller to provide the processor with access to the memory.

[0234] The method embodiments provided in the embodiments of the present application can be executed in a computer terminal, a server or a similar computing device, that is, the above electronic device can include a computer terminal, a server or a similar computing device. Taking running on a server as an example, Fig.10 is a hardware structure block diagram of an electronic device running an exception handling method provided in an embodiment of the present application, such as Fig.10 As shown, the server 1000 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 1010 (the processor 1010 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 1030 for storing data, and one or more storage media 1020 (such as one or more mass storage devices) for storing application programs 1023 or data 1022. Among them, the memory 1030 and the storage medium 1020 can be short-term storage or permanent storage. The program stored in the storage medium 1020 may include one or more modules, each of which may include a series of instruction operations on the server. Furthermore, the central processing unit 1010 can be configured to communicate with the storage medium 1020 and execute a series of instruction operations in the storage medium 1020 on the server 1000. The server 1000 may also include one or more power supplies 1060, one or more wired or wireless network interfaces 1050, one or more input and output interfaces 1040, and / or one or more operating systems 1021, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0235] The input / output interface 1040 may be used to receive or send data via a network. The specific example of the network may include a wireless network provided by a communication provider of the server 1000. In one example, the input / output interface 1040 includes a network adapter (Network Interface Controller, NIC), which may be connected to other network devices through a base station so as to communicate with the Internet. In one example, the input / output interface 1040 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0236] It can be understood by those skilled in the art that Fig.10 The structure shown is only for illustration and does not limit the structure of the above electronic device. Fig.10 More or fewer components as shown, or with Fig.10 Different configurations shown.

[0237] An embodiment of the present application also provides a computer-readable storage medium, which can be set in an electronic device to store at least one instruction or at least one program related to implementing an exception handling method. The at least one instruction or the at least one program is loaded and executed by the processor to implement any one of the exception handling methods provided in the above method embodiments.

[0238] The embodiments of the present application also provide a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of an electronic device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the electronic device executes any one of the exception handling methods provided in the above method embodiments.

[0239] Optionally, in this embodiment, the above-mentioned storage medium may include but is not limited to: a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and other media that can store program codes.

[0240] It should be noted that the above-mentioned sequence of the embodiments of the present application is for description only and does not represent the advantages and disadvantages of the embodiments. The above-mentioned specific embodiments of this specification are described. Other embodiments are within the scope of the attached claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0241] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0242] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0243] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. An exception handling method, characterized in that: The method comprises: In the process of training a deep learning model based on a heterogeneous computing device cluster, abnormal computing power information of the heterogeneous computing device cluster is detected; the abnormal computing power information represents an abnormal heterogeneous computing device with abnormal computing power indicators, and an abnormal computing power indicator corresponding to the abnormal heterogeneous computing device; If the computing power abnormality information is detected, interrupting the training of the deep learning model on the heterogeneous computing device cluster; From the current task processing strategy set, search for a target task processing strategy that matches the abnormal computing power indicator; the task processing strategies in the current task processing strategy set correspond to different computing power indicators, each task processing strategy represents a training resource that needs to be loaded when an abnormality occurs in the corresponding computing power indicator, and the training resource includes the training status information corresponding to the deep learning model when the training is interrupted; If the target task processing strategy is found, the training of the deep learning model on the target heterogeneous computing device cluster is restarted based on the target task processing strategy; the target heterogeneous computing device cluster includes heterogeneous computing devices except the abnormal heterogeneous computing device.

2. The method according to claim 1, characterized in that The method further comprises: If the target task processing strategy is not found, then obtaining task processing configuration information input for the abnormal computing power indicator based on the task processing configuration interface; the task processing configuration information represents the training resources that need to be loaded when the abnormal computing power indicator appears; Restarting the training of the deep learning model on the target heterogeneous computing device cluster based on the task processing configuration information; The task processing configuration information is used as a task processing strategy in the current task processing strategy set, and the current task processing strategy set is updated.

3. The method according to claim 1 or 2, characterized in that: The computing power indicator includes at least one functional indicator, each of which represents a preset operating state of the hardware card; The detecting abnormal computing power information of the heterogeneous computing device cluster includes: For each heterogeneous computing device in the heterogeneous computing device cluster, detecting at least one of the preset operating states on a hardware card in the heterogeneous computing device, and obtaining a state detection result of the heterogeneous computing device corresponding to each of the preset operating states; the state detection result indicates whether the hardware card is in the corresponding preset operating state; If there is a first state detection result in the state detection results corresponding to each of the preset running states of the heterogeneous computing device, it is determined that there is an abnormality in the target functional indicator of the heterogeneous computing device; the first state detection result indicates that the hardware card has a first preset running state corresponding to the first state detection result, and the target functional indicator is a functional indicator corresponding to the first preset running state; Based on the address information of the heterogeneous computing device and the target function indicator, computing power abnormality information of the heterogeneous computing device cluster is generated.

4. The method according to claim 3, characterized in that The computing power indicator includes at least one performance indicator; the detecting of abnormal computing power information of the heterogeneous computing device cluster includes: For each heterogeneous computing device in the heterogeneous computing device cluster, obtaining performance indicator data corresponding to each performance indicator of the heterogeneous computing device; Determining the fluctuation range of each performance indicator of the heterogeneous computing device according to the performance indicator data of each performance indicator of the heterogeneous computing device; Compare the fluctuation range of each performance indicator of the heterogeneous computing device with the preset fluctuation range threshold corresponding to each performance indicator to obtain a performance test result of each performance indicator of the heterogeneous computing device; the performance test result indicates whether the fluctuation range of the corresponding performance indicator exceeds the preset fluctuation range threshold corresponding to the performance indicator; If there is a first performance detection result among the performance detection results corresponding to the performance indicators of the heterogeneous computing device, it is determined that there is an abnormality in the target performance indicator of the heterogeneous computing device; the target performance indicator is the performance indicator corresponding to the first performance detection result, and the first performance detection result indicates that the fluctuation range of the target performance indicator exceeds a preset fluctuation range threshold corresponding to the target performance indicator; Based on the address information of the heterogeneous computing device and the target performance indicator, computing power anomaly information of the heterogeneous computing device cluster is generated.

5. The method according to claim 1 or 2, characterized in that: After interrupting the training of the deep learning model on the heterogeneous computing device cluster, the method further includes: Based on the computing power abnormality information, marking the abnormal heterogeneous computing device in the heterogeneous computing device cluster; Before restarting the training of the deep learning model on the target heterogeneous computing device cluster, the method further includes: A cluster consisting of unmarked heterogeneous computing devices in the heterogeneous computing device cluster is determined as a target heterogeneous computing device cluster.

6. The method according to claim 5, characterized in that The method further comprises: During the training process of the deep learning model, the training state of the deep learning model is recorded at preset time intervals to obtain training state information corresponding to each time point; The restarting the training of the deep learning model on the target heterogeneous computing device cluster based on the target task processing strategy includes: Determine a target time point that is closest to interrupting the training among the time points, and use the training state information corresponding to the target time point as the training state information corresponding to the training resource in the target task processing strategy; Based on the target task processing strategy, restart the training of the deep learning model on the target heterogeneous computing device cluster.

7. An exception handling method, characterized in that: The method comprises: In the process of language processing training of a natural language processing model based on a heterogeneous computing device cluster, abnormal computing power information of the heterogeneous computing device cluster is detected; the abnormal computing power information represents an abnormal heterogeneous computing device with abnormal computing power indicators, and an abnormal computing power indicator corresponding to the abnormal heterogeneous computing device; If the computing power abnormality information is detected, interrupting the language processing training of the natural language processing model on the heterogeneous computing device cluster; From the current task processing strategy set, searching for a target task processing strategy that matches the abnormal computing power indicator; the task processing strategies in the current task processing strategy set correspond to different computing power indicators, and each task processing strategy represents a training resource that needs to be loaded when an abnormality occurs in the corresponding computing power indicator; the training resource includes the training status information corresponding to the natural language processing model when the language processing training is interrupted; If the target task processing strategy is found, the language processing training of the natural language processing model on the target heterogeneous computing device cluster is restarted based on the target task processing strategy; the target heterogeneous computing device cluster includes heterogeneous computing devices except the abnormal heterogeneous computing device.

8. The method according to claim 7, characterized in that The method further comprises: If the target task processing strategy is not found, then obtaining task processing configuration information input for the abnormal computing power indicator based on the task processing configuration interface; the task processing configuration information represents the training resources that need to be loaded when the abnormal computing power indicator appears; Restarting language processing training of the natural language processing model on the target heterogeneous computing device cluster based on the task processing configuration information; The task processing configuration information is used as a task processing strategy in the current task processing strategy set, and the current task processing strategy set is updated.

9. The method according to claim 7 or 8, characterized in that: After interrupting the language processing training of the natural language processing model on the heterogeneous computing device cluster, the method further includes: Based on the computing power abnormality information, marking the abnormal heterogeneous computing device in the heterogeneous computing device cluster; Before restarting the language processing training of the natural language processing model on the target heterogeneous computing device cluster, the method further includes: A cluster consisting of unmarked heterogeneous computing devices in the heterogeneous computing device cluster is determined as a target heterogeneous computing device cluster.

10. The method according to claim 9, characterized in that The method further comprises: During the language processing training process of the natural language processing model, recording the training status of the natural language processing model at preset time intervals to obtain training status information corresponding to each time point; The restarting of the language processing training of the natural language processing model on the target heterogeneous computing device cluster based on the target task processing strategy includes: Determine a target time point that is closest to interrupting the language processing training among the time points, and use the training state information corresponding to the target time point as the training state information corresponding to the training resource in the target task processing strategy; Based on the target task processing strategy, restart the language processing training of the natural language processing model on the target heterogeneous computing device cluster.

11. An exception handling device, characterized in that: The device comprises: A first computing power anomaly detection module is used to detect computing power anomaly information of a heterogeneous computing device cluster during the process of training a deep learning model based on the heterogeneous computing device cluster; the computing power anomaly information represents an abnormal heterogeneous computing device with abnormal computing power indicators, and an abnormal computing power indicator corresponding to the abnormal heterogeneous computing device; A first training interruption module, configured to interrupt the training of the deep learning model on the heterogeneous computing device cluster when the computing power abnormality information is detected; A first processing strategy search module is used to search for a target task processing strategy that matches the abnormal computing power indicator from the current task processing strategy set; the task processing strategies in the current task processing strategy set correspond to different computing power indicators, each task processing strategy represents a training resource that needs to be loaded when an abnormality occurs in the corresponding computing power indicator, and the training resource includes the training state information corresponding to the deep learning model when the training is interrupted; The first training restart module is used to restart the training of the deep learning model on the target heterogeneous computing device cluster based on the target task processing strategy when the target task processing strategy is found; the target heterogeneous computing device cluster includes heterogeneous computing devices other than the abnormal heterogeneous computing device.

12. An exception handling device, characterized in that: The device comprises: A second computing power anomaly detection module is used to detect computing power anomaly information of the heterogeneous computing device cluster during the process of language processing training of the natural language processing model based on the heterogeneous computing device cluster; the computing power anomaly information represents an abnormal heterogeneous computing device with abnormal computing power indicators, and an abnormal computing power indicator corresponding to the abnormal heterogeneous computing device; A second training interruption module is used to interrupt the language processing training of the natural language processing model on the heterogeneous computing device cluster when the computing power abnormality information is detected; A second processing strategy search module is used to search for a target task processing strategy that matches the abnormal computing power indicator from the current task processing strategy set; the task processing strategies in the current task processing strategy set correspond to different computing power indicators, and each task processing strategy represents a training resource that needs to be loaded when the corresponding computing power indicator is abnormal; the training resource includes the training state information corresponding to the natural language processing model when the language processing training is interrupted; The second training restart module is used to restart the language processing training of the natural language processing model on the target heterogeneous computing device cluster based on the target task processing strategy when the target task processing strategy is found; the target heterogeneous computing device cluster includes heterogeneous computing devices except the abnormal heterogeneous computing device.

13. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the processor to implement the exception handling method according to any one of claims 1 to 6 or claims 7 to 10.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the exception handling method according to any one of claims 1 to 6 or claims 7 to 10.

15. A computer program, characterized in that When the computer program is executed by a processor, the exception handling method according to any one of claims 1 to 6 or claims 7 to 10 is implemented.

Images