Model training service processing method and device, equipment and readable storage medium

Abstract

The invention discloses a model training service processing method, apparatus and device, and a readable storage medium. The method comprises the steps of obtaining a service log generated by a model training service; detecting the first training step based on operator execution data of the plurality of set communication operators participating in the first training step in the first training step to obtain a detection result of the first training step; if the detection result of the first training step indicates that the first training step is an abnormal step, determining an abnormal set communication operator from the plurality of set communication operators participating in the first training step based on operator execution data of the plurality of set communication operators participating in the first training step in the first training step; and finding out abnormal equipment from the computing equipment executing the abnormal set communication operator. By adopting the method and the device, the abnormal equipment can be efficiently and accurately detected in the model training business, so that the model training efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, apparatus, device, and readable storage medium for processing a model training business. Background Art

[0002] In model training scenarios, with the rapid development of artificial intelligence technology, the scale of model training continues to expand, and the amount of training data has increased dramatically, placing higher demands on the stability and efficiency of training systems. However, during the actual model training process, due to various factors such as hardware failures, software errors, and network latency, training tasks often encounter problems such as performance degradation and progress stagnation, seriously affecting the model's training efficiency and ultimate training results.

[0003] To ensure the stability and efficiency of the model training system, it is necessary to detect and troubleshoot any anomalies during model training. Operations and maintenance personnel must then isolate or replace these locations. Traditionally, the detection of anomalies during model training primarily relies on a binary search method. This method locates the anomaly by repeatedly splitting a large cluster of computing devices executing model tasks into two, restarting them, and then testing the training tasks. This method continuously reduces the scale of computing device testing, ultimately locating the anomalous device. This detection method requires restarting computing devices, which is time-consuming and inefficient, leading to stagnant model training progress and high idle resource costs. Therefore, in model training, a solution that can efficiently and accurately detect anomalous devices is urgently needed. Summary of the Invention

[0004] The embodiments of the present application provide a method, apparatus, device, and readable storage medium for processing a model training business, which can efficiently and accurately detect abnormal devices in the model training business, thereby improving the efficiency of model training.

[0005] In one aspect, an embodiment of the present application provides a method for processing a model training service. The method is applied to a model training architecture, wherein the model training architecture includes a collective communication library and N computing devices for executing the model training service. The model training service includes W rounds of training steps, and the N computing devices perform communication tasks in each round of the model training service through a collective communication operator provided by the collective communication library. Both N and W are positive integers. The method is applied to a first training step, which refers to any one of the W rounds of training steps. The method includes:

[0006] Obtain the business logs generated by the model training business; the business logs include the communication logs generated by each round of training steps. The communication logs of a round of training steps include multiple collective communication operators participating in the training step, as well as the operator execution data of each collective communication operator in the training step;

[0007] Performing a detection on the first training step based on operator execution data of the plurality of collective communication operators participating in the first training step in the first training step to obtain a detection result of the first training step;

[0008] If the detection result of the first training step indicates that the first training step is an abnormal step, determining an abnormal collective communication operator from the plurality of collective communication operators participating in the first training step based on operator execution data of the plurality of collective communication operators participating in the first training step in the first training step;

[0009] Find abnormal devices from computing devices that execute abnormal set communication operators.

[0010] In one aspect, an embodiment of the present application provides a device for processing a model training service. The device is applied to a model training architecture, wherein the model training architecture includes a collective communication library and N computing devices for executing the model training service. The model training service includes W rounds of training steps, and the N computing devices perform communication tasks in each round of the model training service using a collective communication operator provided by the collective communication library. N and W are both positive integers. The device is applied to a first training step, which refers to any one of the W rounds of training steps. The device includes:

[0011] The log acquisition module is used to obtain the business logs generated by the model training business; the business logs include the communication logs generated by each round of training steps. The communication logs of a round of training steps include multiple collective communication operators participating in the training step, as well as the operator execution data of each collective communication operator in the training step;

[0012] a step detection module, configured to detect the first training step based on operator execution data of the plurality of collective communication operators participating in the first training step in the first training step, and obtain a detection result of the first training step;

[0013] an operator determination module configured to, if the detection result of the first training step indicates that the first training step is an abnormal step, determine an abnormal collective communication operator from the multiple collective communication operators participating in the first training step based on operator execution data of the multiple collective communication operators participating in the first training step in the first training step;

[0014] The device search module is used to find abnormal devices from computing devices that execute abnormal set communication operators.

[0015] In one embodiment, the operator execution data of each collective communication operator in a round of training steps includes the execution time of the collective communication operator in the training step;

[0016] The step detection module detects the first training step based on operator execution data of multiple collective communication operators participating in the first training step in the first training step, and obtains a specific implementation method of the detection result of the first training step, including:

[0017] Among the multiple collective communication operators participating in the first training step, the step duration of the first training step is calculated based on the execution time of each collective communication operator in the first training step;

[0018] Obtain the duration of the second training step; the second training step refers to the training step other than the first training step in the W rounds of training steps;

[0019] Sort the duration of the first training step and the duration of the second training step in order of size to obtain a duration sequence corresponding to W rounds of training steps;

[0020] Based on the duration sequence and the step duration of the first training step, the first training step is detected to obtain a detection result of the first training step.

[0021] In one embodiment, the step detection module detects the first training step based on the duration sequence and the step duration of the first training step, and a specific implementation method for obtaining the detection result of the first training step includes:

[0022] The probability distribution fitting function is used to perform probability distribution fitting on the time series to obtain the probability distribution corresponding to the time series;

[0023] Solving the probability distribution according to a preset quantile to obtain a solution result of the probability distribution for the preset quantile;

[0024] If the duration of the first training step is longer than the solution result, the detection result of the first training step is determined to be a failed detection result; the failed detection result indicates that the first training step is an abnormal step;

[0025] If the step duration of the first training step is less than the solution result, the detection result of the first training step is determined to be a pass detection result; the pass detection result indicates that the first training step is a non-abnormal step.

[0026] In one embodiment, the operator execution data of each collective communication operator in a round of training steps includes the execution time of the collective communication operator in the training step;

[0027] The operator determination module determines a specific implementation of an abnormal collective communication operator from the multiple collective communication operators participating in the first training step based on operator execution data of the multiple collective communication operators participating in the first training step in the first training step, including:

[0028] sorting the plurality of collective communication operators participating in the first training step according to their respective execution times in the first training step to obtain a target sequence;

[0029] Preprocessing operator execution data of each collective communication operator in the target sequence in the first training step to obtain step features of the first training step;

[0030] Calling an operator detection model based on the step features of the first training step;

[0031] The step features of the first training step are analyzed using the model parameters of the operator detection model, and an abnormal set communication operator having abnormal execution time is output.

[0032] In one embodiment, the operator determination module preprocesses the operator execution data of each collective communication operator in the target sequence in the first training step to obtain a specific implementation of the step feature of the first training step, including:

[0033] Obtain the full set of collective communication operators provided by the collective communication library, and perform sequence clipping on the target sequence according to the full set of collective communication operators to obtain a clipped sequence corresponding to the target sequence; the clipped sequence includes any collective communication operator in the full set of collective communication operators;

[0034] Based on the execution time of each collective communication operator in the trimmed sequence in the first training step, feature engineering processing is performed on the operator execution data of each collective communication operator in the trimmed sequence to obtain the execution features corresponding to each collective communication operator;

[0035] Assembling the operator execution data and corresponding execution features of each collective communication operator in the trimmed sequence in the first training step to obtain the operator assembly features of each collective communication operator in the trimmed sequence;

[0036] The operator assembly features of each collective communication operator in the clipping sequence are fused to obtain the step features of the first training step.

[0037] In one embodiment, the training process of the operator detection model includes:

[0038] Obtain a sample communication log for a sample training step; a sample training step refers to a round of training steps in a sample model training service. The sample communication log includes multiple sample set communication operators participating in the sample training step, as well as operator execution data of each sample set communication operator in the sample training step; the operator execution data of each sample set communication operator in the sample training step includes the execution time of the sample set communication operator in the sample training step;

[0039] Sort the multiple sample set communication operators by their execution times in the sample training steps to obtain a sample sequence;

[0040] Preprocessing the operator execution data of each sample set communication operator in the sample sequence to obtain the step features of the sample training step;

[0041] Based on the step features of the sample training steps, the initial operator detection model is called, and the step features of the sample training steps are analyzed using the model parameters of the initial operator detection model to output the communication operators of the abnormal sample set with abnormal execution time.

[0042] According to the difference between the communication operator labels of the sample set corresponding to the sample training step and the communication operator of the abnormal sample set, the initial operator detection model is trained to obtain the operator detection model.

[0043] In one embodiment, the operator execution data of each collective communication operator in a round of training steps includes at least one computation processor executing the collective communication operator in the training steps;

[0044] The specific implementation method of the device search module to find abnormal devices from the computing devices that execute the abnormal set communication operator includes:

[0045] Determine the plurality of collective communication operators participating in the first training step as step participating operators;

[0046] Creating a topology graph for a first training step based on at least one computational processor executing each step participant operator; the topology graph comprising a plurality of operator topology graphs, one operator topology graph corresponding to each step participant operator, one operator topology graph comprising at least one node, one node in the operator topology graph corresponding to a computational processor of the step participant operator corresponding to the operator topology graph;

[0047] Obtain the probability transfer matrix corresponding to the topology graph; the probability transfer matrix includes the probability that any two nodes in the topology graph transfer to each other;

[0048] Moving from a first node to a second node in the topology graph according to the probability transfer matrix, obtaining a frequency at which each node in the topology graph is visited during the movement;

[0049] According to the access frequency of each node in the topology graph during the movement, abnormal devices are found from N computing devices.

[0050] In one embodiment, a specific implementation of the device search module creating a topology map for the first training step based on at least one computing processor participating in the operator in each step includes:

[0051] Count the number of at least one computing processor executing each step participating operator, and create T nodes for each step participating operator according to the number of at least one computing processor executing each step participating operator; T is the number of at least one computing processor executing the step participating operator, and one node in the T nodes corresponds to a computing processor executing the step participating operator;

[0052] Establish connection edges between the T nodes corresponding to the participating operators in each step, and obtain the operator topology graph corresponding to the participating operators in each step;

[0053] Aggregate the operator topology graphs corresponding to the participating operators in each step of the first training step to obtain an initial topology graph of the first training step;

[0054] A corresponding node weight is determined for each node in the initial topological graph to obtain a topological graph of the first training step.

[0055] In one embodiment, the specific implementation method of the device search module determining the corresponding node weight for each node in the initial topology graph includes:

[0056] Determine any node in the initial topology graph as the target node, and determine the step participating operator corresponding to the target node as the target operator;

[0057] Determine the weight calculation method of the target node based on the operator type of the target operator;

[0058] The node weight corresponding to the target node is calculated using the target node's weight calculation method.

[0059] In one embodiment, the device search module determines a specific implementation of a weight calculation method for a target node based on the anomaly type of the target operator, including:

[0060] If the operator type of the target operator is an abnormal type, the weight calculation method of the target node is determined to be the first calculation method; the first calculation method is used to calculate the execution time, resource usage and abnormality of the target operator;

[0061] If the operator type of the target operator is a non-exceptional type, the weight calculation method of the target node is determined to be the second calculation method; the second calculation method is used to calculate the execution time and resource usage of the target operator.

[0062] In one embodiment, a specific implementation method of the device search module for finding abnormal devices from N computing devices based on the access frequency of each node in the topology map during movement includes:

[0063] Get the maximum frequency among the frequencies at which each node in the topology graph is visited during movement;

[0064] Determine the node indicated by the maximum frequency as an abnormal node;

[0065] The computing processor corresponding to the abnormal node is determined as the abnormal processor, and the computing device running the abnormal processor among the N computing devices is determined as the abnormal device.

[0066] In one aspect, an embodiment of the present application provides a computer device, including: a processor and a memory;

[0067] The memory stores a computer program, and when the computer program is executed by the processor, the processor executes the method in the embodiment of the present application.

[0068] On one hand, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. The computer program includes program instructions. When the program instructions are executed by a processor, the method in the embodiment of the present application is executed.

[0069] In one aspect of the present application, a computer program product is provided, comprising a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the method provided in one aspect of the embodiments of the present application.

[0070] In an embodiment of the present application, a solution is provided for detecting abnormal devices in a model training business based on a collective communication log, which can efficiently and accurately detect abnormal devices without interrupting or restarting the model training business. Specifically, the collective communication log in the present application may refer to a log for a collective communication library. Since each computing device in the model training business needs to use a collective communication library to perform communication tasks, the present application can monitor the entire collective communication of each computing device. A model training business may include multiple rounds of training steps, and the present application can record the communication logs collected from each computing device in each round of training steps in the model training business through log monitoring. The communication log of a round of training steps includes multiple collective communication operators participating in the training steps, as well as the operator execution data of each collective communication operator in the training step; in this way, subsequent monitoring of any For a round of training steps, the operator execution data of each collective communication operator participating in this training step in this round of training steps can be used to detect the training step to detect whether it is an abnormal step; after detecting the abnormal step, the abnormal collective communication operator can be found from the collective communication operators participating in this abnormal step based on the operator execution data of each collective communication operator participating in this abnormal step. In this way, the abnormal step can be accurately located to a certain operator in the step; after finding the abnormal collective communication operator, the abnormal device can be found based on the computing device that executes the abnormal collective communication operator. It can be seen that this application can first detect the abnormal training step based on the communication log of each round of training steps in the model training business, then detect the abnormal collective communication operator from the training step, and then find the abnormal device from the computing device that executes the abnormal collective communication operator. In this way, by narrowing the search scope step by step, the abnormal device can be found efficiently and accurately without interrupting or restarting the model training business, so as to realize equipment operation and maintenance, ensure the smooth execution of the model training business, and improve the efficiency of model training. In summary, this application can efficiently and accurately detect abnormal devices in model training business, thereby improving model training efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0072] Figure 1 This is a schematic diagram of an architecture of a model training system provided by an exemplary embodiment of the present application;

[0073] Figure 2 This is a schematic diagram of a model training architecture provided in an embodiment of the present application;

[0074] Figure 3 This is a flowchart of a method for processing a model training service provided by an exemplary embodiment of the present application;

[0075] Figure 4 This is a flow chart of searching for abnormal devices based on an abnormal set communication operator provided by an embodiment of the present application;

[0076] Figure 5 This is a schematic diagram of creating an initial topology diagram provided by an embodiment of the present application;

[0077] Figure 6 This is a structural diagram of a processing device for a model training service provided in an embodiment of the present application;

[0078] Figure 7 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0079] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0080] The embodiments of the present application involve relevant technologies in the model training process. For ease of understanding, the relevant technical terms and concepts involved in the embodiments of the present application will be briefly explained below.

[0081] Iteration: A repeated step in model training. A training step can be understood as a round of training. Each training step includes forward propagation, backpropagation, and weight update.

[0082] NVIDIA Collective Communications Library (NCCL): A communications library for high-performance distributed computing that provides a range of important concepts and functionality. This library includes multiple collective communication operators, such as AllReduce, Broadcast, Reduce, and AllGather, which enable efficient collective communication operations across a graphics processing unit (GPU) cluster.

[0083] Operator: A mathematical formula or algorithm used to perform specific operations in model training. One operator corresponds to one operation and can include computational operators and communication operators. Common computational operators in model training include convolution, fully connected, pooling, activation functions (such as ReLU and Sigmoid), and batch normalization. Common communication operators in model training include the collective communication operators mentioned above, specifically including broadcast, all-reduce, gather, and all-gather operators.

[0084] Slow node problem: The slow node problem in the model training business mainly refers to the problem that during the training process, the processing speed of some nodes (the nodes in this application can be understood as computing devices) is slow, resulting in a decrease in the overall training efficiency. This problem is particularly common in large-scale model training because it affects the progress and effect of the entire training process. The main reasons for the slow node problem may include the following four points: 1. Insufficient computing power. Some nodes may have slow processing speeds due to low hardware configuration; 2. Memory bandwidth limitations. Insufficient memory bandwidth of some nodes will lead to slow data transmission speeds, affecting the processing efficiency of the nodes; 3. Communication delays; 4. Slow data loading.

[0085] In practical applications, a model training process typically includes multiple training rounds, each executed by a cluster of computing devices. When training large models, the scale can reach thousands or even tens of thousands of cards. However, large-scale model training is not a stable process, and can often experience abnormal training steps. For example, performance jitter and throughput degradation can occur during training steps. These abnormalities are generally caused by slow nodes and do not interrupt training (no abnormality alerts are generated), but they can persistently impact training progress and results. If a model training process experiences stalled progress or performance degradation, it is necessary to detect and locate slow nodes in each training round to identify the abnormal computing device. This allows subsequent operations and maintenance personnel to isolate or replace the device to ensure smooth training. Traditionally, the detection of abnormal devices during model training primarily relies on a binary approach. This method involves repeatedly splitting a large cluster of computing devices performing model training into two, restarting them, and then retesting the training task. This approach continuously reduces the number of computing devices tested, ultimately locating the abnormal device. However, this method requires constant restarts of computing devices, which can stall the overall progress of model training, resulting in time-consuming and inefficient results.

[0086] In order to efficiently and accurately detect abnormal devices in the model training business and improve the efficiency of model training, the present application provides a solution for detecting abnormal devices in the model training business based on collective communication logs. Abnormal devices can be detected efficiently and accurately without interrupting or restarting the model training business. Specifically, the solution provided by the present application for detecting abnormal devices in the model training business based on collective communication logs mainly includes the following four consecutive steps: 1. Obtain the business logs generated in the model training business. The business log here can be obtained by real-time monitoring and recording of the collective communication operations in the model training business. In the model training business, in addition to using computing operators to perform computing tasks, each computing device also needs to use communication operators to perform communication tasks. For the communication operators in the model training business, the collective communication operators provided by the collective communication library NCCL are usually used. Each computing device can use these collective communication operators to complete different collective communication operations (for example, broadcast operations, full protocol operations, etc.). Based on this, the present application can monitor the entire collective communication to obtain which collective communication operators are executed by the computing device in each round of training steps and which execution data are generated when executing each collective communication operator. The collective communication operators executed by the computing device in a round of training steps can be understood as the collective communication operators participating in this round of training steps. In a round of training steps, the execution data generated by any collective communication operator participating in this round of training steps can be called the operator execution data of the collective communication operator in this training step. In this way, the operator execution data of each collective communication operator participating in a round of training steps can constitute the communication log of the training step, and the communication log of each round of training steps can constitute the business log of the entire model training business; 2. For any round of training steps, the training step can be detected based on the operator execution data of each collective communication operator participating in the training step to analyze whether the training step is abnormal; 3. After detecting whether each training step is abnormal, the abnormal training step can be called an abnormal step, and for any abnormal step, the operator execution data of each collective communication operator participating in the abnormal step in the abnormal step can be used to analyze and detect each collective communication operator participating in the abnormal step to determine whether these operators are abnormal; 4. After finding the abnormal collective communication operator, the computing device that executes the abnormal collective communication operator can be obtained. Compared with the full number of computing devices, the number of computing devices that execute the abnormal collective communication operator in the abnormal step is extremely small, so that the abnormal device can be quickly found.

[0087] It can be seen that through the solution provided in the present application, by adding a logging function to the collective communication of the computing device without interrupting or restarting the computing device, it is possible to quickly and accurately detect and locate abnormal steps and abnormal collective communication operators in abnormal steps based on the collective communication log, thereby realizing the root cause location of the slow node based on the abnormal collective communication operator, finding the computing device with the abnormality, and facilitating the operation and maintenance personnel to repair or replace the abnormal equipment, thereby improving the overall training efficiency of the model training business.

[0088] The solution provided in the embodiments of the present application can be applied to any application scenario that requires an artificial intelligence model to process relevant business data, including but not limited to short video push scenarios, game scenarios, etc.

[0089] The short video push scenario can refer to a scenario in which video data is continuously pushed to users. Users can request to update the display of the next video data by performing operations to pull video data (such as sliding the video display interface of the terminal device). In the short video push scenario, users can continuously refresh and browse different video data by continuously performing operations to pull video data. In the short video push scenario, the video data pushed to users can be predicted by the video recommendation model. In the training business of the video recommendation model, this solution can be used to perform real-time detection of the training process, and abnormal equipment can be found in a timely manner for repair or replacement.

[0090] A gaming scenario can refer to a user engaging in an immersive game through a terminal device. In this scenario, when a user initiates a game, an intelligent model can match the user's allies with the enemy. During the training of this intelligent model, this solution can be used to monitor the training process in real time, identifying any anomalies and promptly repairing or replacing them.

[0091] To sum up, the solution provided by the embodiment of the present application can efficiently and accurately detect abnormal devices in the model training business, thereby improving the model training efficiency and effectively improving the business coverage to a certain extent (such as expanding the applicable scenarios).

[0092] It should be noted that the several application scenarios given above are only examples and do not limit the application scenarios to which the solutions provided in the embodiments of the present application are applicable.

[0093] Furthermore, the solution provided in the embodiment of the present application can be executed by a computer device, which can be used to monitor and manage the model training business. The computer device may include a terminal or a server, or a terminal and a server. To facilitate understanding of the solution provided in the embodiment of the present application, the following is combined with Figure 1The model training system shown introduces the application scenarios involved in the embodiments of the present application; wherein, Figure 1 This is a schematic diagram of the architecture of a model training system provided by an exemplary embodiment of the present application. Figure 1 As shown, the system includes a terminal 101 and a server 102; wherein:

[0094] 1) The terminal device may include the terminal device used by the user. Depending on the application scenario and field to which the present solution is applied, the terminal providing the solution provided by the embodiment of the present application may be different. The terminal device may include, but is not limited to: a smartphone (such as a smartphone deploying the Android system, or a smartphone deploying the Internetworking Operating System (IOS)), a tablet computer, a portable personal computer, a mobile Internet device (MID), a vehicle-mounted device, a head-mounted device, a smart home, and an intelligent voice interaction device. The embodiment of the present application does not limit the type of terminal device, which is explained here.

[0095] For example, in the short video push scenario, the terminal device can be a smart phone; that is, in this implementation method, the solution provided by the embodiment of the present application can be deployed on the smart phone; when the user uses the short video push application on the smart phone, the smart phone can pre-train the video recommendation model based on the business data obtained in the short video push application (for example: historical video data watched, liked, and collected by a user in a certain time period). During the model training process, this solution can be used to detect the training process, so as to find abnormal equipment in time and repair or replace it. For another example, in the smart car scenario, the application deployed with the solution provided by the embodiment of the present application is a car application; the types of the car application may include but are not limited to: music, video, or games, etc.

[0096] Applications can be computer programs designed to perform one or more specific tasks. By categorizing applications according to different dimensions (such as their operating mode and functionality), we can identify the types of the same application across different dimensions. For example, based on their operating mode, applications may include, but are not limited to, clients installed on terminals, mini-programs (subprograms of clients) that can be used without downloading or installing, and World Wide Web (Web) applications opened via a browser. Another example is based on their functional type, applications may include, but are not limited to, instant messaging (IM) applications, content interaction applications, audio applications, or video applications. IM applications refer to internet-based applications for instant messaging and social interaction. They may include, but are not limited to, applications with communication functionality, map applications with interactive functionality, and gaming applications. Content interaction applications refer to applications that enable content interaction, such as sharing platforms, personal spaces, and news applications. Audio applications refer to internet-based applications that implement audio functionality. Audio applications may include, but are not limited to, music applications with music playback and editing capabilities, radio applications with radio playback capabilities, or live streaming applications with live streaming capabilities. Video applications refer to applications that can play images. Video applications may include but are not limited to: applications with short videos (video length is often short, such as a few seconds or minutes, etc.) (such as short video push applications), applications with long videos (such as videos with long playback time such as movies or TV series), etc.

[0097] Of course, the solution provided in the embodiments of the present application can be directly deployed on a device (such as a smartphone), a server (such as a backend server), or deployed outside of an application, as described above, or can be deployed in a device or application in the form of a plug-in. The embodiments of the present application do not limit the carrier of the deployment solution.

[0098] 2) The server can be a server corresponding to the terminal, which is used to interact with the terminal for data to provide computing and application service support for the terminal. Specifically, the server is a background server corresponding to the application deployed in the terminal, which is used to interact with the terminal to provide computing and application servers for the application. Among them, the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0099] For example, in the short video push scenario, the terminal device can be a server; that is, in this implementation method, the solution provided by the embodiment of the present application can be deployed on the server; when a user uses a short video push application on a smartphone, the smartphone can obtain relevant business data in the short video push application, such as: historical video data watched, liked, and collected by a user in a certain time period, and then send these business data to the server, which trains the video recommendation model based on these business data. During the model training process, this solution can be used to detect the training process, so as to find abnormal equipment in time and repair or replace it.

[0100] The terminal 101 and the server 102 may be connected directly or indirectly via wired or wireless communication, which is not limited in this application. In addition, the embodiment of this application does not limit the number of terminals and servers; Figure 1 The number of terminals 101 and servers 102 is only one for example. In actual applications, multiple distributed servers may be included, which is specially explained here.

[0101] Based on the solution and system architecture described above, the following points need to be explained:

[0102] ① The above-mentioned embodiments of this application Figure 1The system shown is for the purpose of more clearly illustrating the technical solution of the embodiment of the present application, and does not constitute a limitation on the technical solution provided by the embodiment of the present application. It is known to those skilled in the art that with the evolution of the system architecture and the emergence of new business scenarios, the technical solution provided by the embodiment of the present application is also applicable to similar technical problems. For example, the above is an introduction to an application scenario of the present solution by taking the example of the execution subject "computer device" of the embodiment of the present application including a terminal and a server, that is, the terminal and the server jointly execute the solution provided by the embodiment of the present application as an example; it should be understood that in actual applications, the computer device can also be a terminal or a server, that is, it supports the terminal or the server to execute the solution provided by the embodiment of the present application alone.

[0103] ② The collection and processing of relevant data in the embodiments of this application should be strictly in accordance with the requirements of relevant laws and regulations. The acquisition of personal information must be subject to the knowledge or consent of the individual subject (or the legal basis for obtaining the information), and subsequent data use and processing must be carried out within the scope of authorization of laws and regulations and the subject of personal information. For example, when the embodiments of this application are applied to specific products or technologies, such as obtaining the performance data of a user in a certain game, the user's permission or consent must be obtained, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant region.

[0104] Based on the above-described solution, the present application embodiment proposes a more detailed method for processing model training services. This method for processing model training services can be applied to model training services. For ease of understanding, the following briefly introduces the system architecture of the model training service. Please refer to Figure 2 , Figure 2 This is a schematic diagram of a model training architecture provided in an embodiment of the present application. Figure 2 As shown, the architecture includes a computing device cluster 201, a terminal device 202 and a collective communication library 203; wherein:

[0105] 1) The computing device cluster 201 may include multiple (e.g., N, where N is a positive integer) computing devices, each of which may be a server. These computing devices may train a model based on the business data provided by the terminal device 202 to improve the accuracy and rationality of the model's output.

[0106] 2) Terminal device 202 can refer to the aforementioned terminal 101. Any application (e.g., social applications, gaming applications, short video push applications, etc.) can be deployed on this terminal device 202. Terminal device 202 can collect business data from any application and send it to computing device cluster 201. Computing device cluster 201 then trains the model corresponding to the application based on this business data. The trained model can then be put into use in the application.

[0107] 3) The collective communication library 203 can provide different collective communication operators. When the computing device cluster 201 trains a model, it can use the collective communication operators in the collective communication library 203 to perform different collective communication operations. The collective communication library can preferably be the above-mentioned NCCL.

[0108] Based on the above-described scheme and model training architecture, the processing method of the model training business proposed in the embodiment of the present application will be introduced in detail with reference to the accompanying drawings.

[0109] See Figure 3 , Figure 3 This is a flow chart of a method for processing a model training service provided by an exemplary embodiment of the present application. This flow chart may refer to the flow chart of a solution for detecting abnormal devices in a model training service based on aggregated communication logs provided by an embodiment of the present application. This method for processing a model training service may be executed by a computer device in the aforementioned system, such as a terminal and / or server. This method for processing a model training service may include at least the following steps S301-S304:

[0110] Step S301, obtain the business log generated by the model training business; the business log includes the communication log generated by each round of training steps, and the communication log of a round of training steps includes multiple collective communication operators participating in the training steps, and the operator execution data of each collective communication operator in the training step.

[0111] In this application, the model training service refers to the training service for any model, and a training service may include multiple rounds of iterative training steps (hereinafter, the model training service including W rounds of training steps will be used as an example for explanation, where W can be a positive integer). The model training service can be executed by a computing device cluster, which may include N computing devices. These N computing devices will perform forward propagation, backward propagation, and model weight update steps in each round of training steps. In the forward propagation, backward propagation, or weight update steps of each round of training steps, different computing devices will perform calculations and communications to complete this forward propagation, backward propagation, or weight update step. For the calculation of the computing device, the computing device may use computing operators to complete different computing operations, and for the communication of the computing device, different computing devices may use communication operators to complete different communication operations.

[0112] This application can specifically use the collective communication operators provided in the collective communication library to perform communication operations. That is to say, in the model training business of this application, N computing devices can perform different collective communication operations through the various collective communication operators provided by the collective communication library, thereby completing the communication task. A collective communication operator can correspond to a collective communication operation (for example, a broadcast operator can correspond to a broadcast operation), and a collective communication operation can be completed by one or more computing devices among the N computing devices. Under normal circumstances, the N computing devices will execute the full amount of collective communication operators provided by the collective communication library in each round of training steps to perform different collective communication operations in this round of training steps, and some collective communication operations need to be executed multiple times before and after, then the collective communication operators corresponding to these collective communication operations will be executed multiple times before and after accordingly. In general, the N computing devices in this application will perform forward propagation, backward propagation or weight update steps in each round of training steps. In the forward propagation, backward propagation or weight update steps, the N computing devices will perform different collective communication operations through the various collective communication operators provided by the collective communication library. The collective communication operator executed by the computing device in a round of training steps can be understood as the collective communication operator participating in this round of training steps. A collective communication operator participating in this round of training steps is executed by one or more computing devices among the N computing devices.

[0113] Based on the fact that each computing device executes each collective communication operator in each round of training steps, this application develops a solution for monitoring collective communication, which can directly monitor the collective communication data in the computing device in real time to obtain the logs generated by the collective communication process of each round of training steps in real time. Specifically, when any computing device executes a collective communication operator, the specific process is: calling the central processing unit (CPU) running on the computing device, and the CPU generates an instruction, which is sent to the graphics processor running on the computing device. The graphics processor generates a processing process based on this instruction to perform the task indicated by the instruction based on this process.

[0114] It can be seen that the execution of a collective communication operator can be understood as being jointly executed by the processes of the graphics processors in different computing devices. Based on this, the present application can monitor and record which collective communication operators each GPU participates in executing in each round of training steps, as well as the execution data generated when participating in the execution of each collective communication operator. For each collective communication operator that a GPU participates in executing in a round of training steps, corresponding execution data will be generated (execution data such as: the time the GPU executes the collective communication operator, the operation type to which the collective communication operator belongs (this operation type can be defined based on the function implemented by the collective communication operator, for example, if the collective communication operator is a broadcast operator and the function it implements is a broadcast function, then the operation type to which the collective communication operator belongs can be a broadcast type), the time when the execution of the collective communication operator ends, etc.). After monitoring and collecting the execution data of each collective communication operator participated in executing in a round of training steps, since different GPUs may participate in the execution of the same collective communication operator at the same time, the present application can merge the execution data of different GPUs of a collective communication operator, thereby forming the operator execution data of the collective communication operator in a round of training steps.

[0115] For example, in a certain round of training steps, each computing device executes 6 collective communication operators. For a certain collective communication operator a, the GPUs involved in its execution include GPU0 and GPU1. Then, this application obtains the execution data 0 generated by monitoring and collecting GPU0 when executing the collective communication operator a in this round of training steps, and can monitor and collect the execution data 1 generated by GPU1 when executing the collective communication operator a in this round of training steps. By merging the execution data 0 with the execution data 1, the operator execution data of the collective communication operator a in this round of training steps can be obtained. For a round of training steps, the number of collective communication operators involved in its execution is usually not one. Then, this application can obtain the operator execution data of each collective communication operator in this round of training steps, and merge (for example, splice) the operator execution data of these collective communication operators in this round of training steps to obtain the communication log of this round of training steps; by merging the communication logs of each round of training steps of the model training business, the business log of this model training business can be obtained.

[0116] To sum up, this application has developed a monitoring solution for collective communication in model training business, which can monitor and collect the communication logs generated by each round of training steps during the collective communication process in real time. The communication log may include the number of processes of each collective communication operator participating in this round of training steps, the operation type of the operator, the execution start time and end time of the operator, and other elements. These detailed log information will provide strong support for subsequent step detection, operator detection and abnormal device detection.

[0117] Step S302 : Based on operator execution data of a plurality of collective communication operators participating in the first training step in the first training step, the first training step is detected to obtain a detection result of the first training step.

[0118] In this application, after obtaining the communication log of each round of training steps in the model training business, the communication log can be processed and analyzed to detect whether each round of training steps is abnormal. Based on the above, it can be seen that the collective communication operator in a round of training steps will include the execution time of the operator in the round of training steps (for example, the start execution time and the end execution time) in the operator execution data of the round of training steps. The detection of any round of training steps can be based on the operator execution data of each collective communication operator participating in the training step in the round of training steps. Taking the first training step (any training step in the W rounds of training steps) as an example, the specific implementation process of detecting the first training step based on the operator execution data of multiple collective communication operators participating in the first training step to obtain the detection result of the first training step may at least include: obtaining multiple collective communication operators participating in the first training step, extracting the execution time of each collective communication operator in the first training step from the operator execution data of the multiple collective communication operators participating in the first training step, and calculating the step duration of the first training step based on the execution time of each collective communication operator in the first training step. The specific process of counting the step duration of the first training step may include the following three consecutive steps: 1. According to the starting execution time of each collective communication operator participating in the first training step in the first training step, these collective communication operators are sorted in order from early to late to obtain an operator sequence; 2. The collective communication operator located at the end position of the sequence in the operator sequence is used as the end operator, and the collective communication operator located at the start position of the sequence is used as the start operator. The end execution time of the end operator can be subtracted from the start execution time of the start operator to obtain a difference duration; 3. Based on the end execution time and start execution time of the above-mentioned start operator, the operator execution duration of the start operator in this first training step is calculated, and then the above-mentioned difference duration is summed with the operator execution time. The sum result can be used as the step duration of the first training step.

[0119] Except for the first training step, for the other training steps in the W rounds of training steps (the training steps in the W rounds of training steps except the first training step, any training step in the training steps except the first training step may be referred to as the second training step in this application), the same method is used to statistically calculate their step durations. That is, the step duration of the second training step can be statistically calculated based on the operator execution data of each collective communication operator participating in the second training step in the second training step. After obtaining the step duration of each second training step, the step duration of each round of training steps in the W rounds of training steps has been obtained. Based on the step duration of all training steps, it is possible to analyze whether the step duration of a particular round of training steps is abnormal.

[0120] Specifically, the step duration of the first training step and the step duration of the second training step can be sorted in order of size. For example, the step duration of W rounds of training steps can be sorted from large to small or from small to large. The sequence obtained after sorting can be called the duration sequence corresponding to the W rounds of training steps; based on this duration sequence and the step duration of any round of training steps itself, the round of training steps can be tested to obtain the test results of the round of training steps. Taking the first training step as an example, the specific implementation process of detecting the first training step based on the duration of the time sequence and the step duration of the first training step and obtaining the detection result of the first training step may at least include: using a probability distribution fitting function to perform probability distribution fitting processing on the time sequence to obtain a probability distribution corresponding to the time sequence; the probability distribution fitting function here can be any probability distribution function, for example, a normal distribution function, a Poisson distribution function, etc., and the time sequence can be fitted into a probability distribution (for example, a normal distribution, a Poisson distribution, etc.) through the probability distribution fitting function; after determining the probability distribution corresponding to the time sequence, a tail delay boundary can be set for the step duration of each round of training steps through this probability distribution, and this tail delay boundary can be used as a duration upper limit value. Subsequently, based on this tail delay boundary, whether any round of training steps is an abnormal step can be detected. The specific implementation method of detecting whether any round of training steps is an abnormal step based on this tail delay boundary can be shown as formula (1):

[0121] Exception(x i )= i > pf(λ) Formula (1)

[0122] Where, as shown in formula (1), x i It can be used to characterize the duration of any round of training steps, and Ppf(λ) can be used to characterize the tail delay boundary; Exception(x i) can be used to represent the comparison between the step duration of the training step and the tail delay boundary. Based on this formula (1), once the step duration of a training step in a round exceeds the tail delay boundary, it can be considered that the step duration of the training step in the round is too long and there is an abnormality.

[0123] In a specific implementation, the tail delay bound can be set to a preset quantile of the probability distribution. This preset quantile can be set based on actual business needs. For example, it can be set to 99%, 75%, and so on. If the tail delay bound is set to the 99th quantile of the probability distribution, then the tail delay bound is the 99th percentile value of the probability distribution. To facilitate understanding, the following is a brief introduction to the concept of quantiles of probability distribution. For example, assuming there are 1,000 numbers (positive numbers) in a probability distribution, the 5%, 30%, 50%, 70%, and 99% quantiles of these numbers are 3.0, 5.0, 6.0, 9.0, and 12.0, respectively. This means that 5% of the numbers are distributed between 0 and 3.0, 25% of the numbers are distributed between 3.0 and 5.0, 20% of the numbers are distributed between 5.0 and 6.0, 20% of the numbers are distributed between 6.0 and 9.0, 29% of the numbers are distributed between 9.0 and 12.0, and 1% of the numbers are greater than 12.0. This is the statistical understanding of quantiles. Therefore, if you want to solve the quantile of a number in a group of numbers, you only need to sort the group of numbers, then count the number of numbers that are less than or equal to the number, and then divide it by the total number of numbers included in the group of numbers. Conversely, if the preset quantile is known, you need to solve the value corresponding to the quantile by reverse deduction. In this way, after obtaining the probability distribution corresponding to the time sequence, you can solve the probability distribution according to the preset quantile to obtain the solution result of the probability distribution for the preset quantile (that is, the value corresponding to the quantile); the solution result can be used as the duration upper limit of the step duration. If the step duration of the first training step is greater than the solution result, it can be determined that the detection result of the first training step is a failed detection result; the failed detection result indicates that the first training step is an abnormal step; and if the step duration of the first training step is less than the solution result, it can be determined that the detection result of the first training step is a passed detection result; the passed detection result indicates that the first training step is a non-abnormal step.

[0124] Based on the above, it can be seen that in the model training business, anomaly detection of training steps is a key link in timely detection of abnormal equipment. This application is based on the operator execution data of each collective communication operator participating in the training step in the training step, and can extract the execution time of each collective communication operator in the training step, that is, the execution time can be statistically calculated to obtain the step duration of the training step. According to the step duration of each round of training steps, a reasonable tail delay boundary can be set. Based on this tail delay boundary, training steps whose step duration deviates from the normal range can be effectively identified. These training steps can be regarded as training steps with abnormal duration. The abnormal equipment is likely to have an abnormality when executing this training step. Subsequently, the relevant data in this training step can be focused on for investigation and detection.

[0125] Step S303: If the detection result of the first training step indicates that the first training step is an abnormal step, an abnormal collective communication operator is determined from the multiple collective communication operators participating in the first training step based on the operator execution data of the multiple collective communication operators participating in the first training step in the first training step.

[0126] In the present application, after detecting an abnormal training step, each collective communication operator involved in the abnormal step can be tested to determine which collective communication operator was executed during the abnormality. The detection of collective communication operators can also be performed based on the operator execution data of each collective communication operator involved in the training step. The specific implementation process can at least include: sorting the multiple collective communication operators involved in the first training step based on their execution time in the first training step, and the resulting sequence can be called a target sequence; preprocessing the operator execution data of each collective communication operator in the target sequence in the first training step to obtain the step characteristics of the first training step; the preprocessing here can include the following two steps: 1. Sequence clipping. Since the number of operators involved in a round of training steps is huge, analyzing the operator execution data of all collective communication operators is very time-consuming and labor-intensive. In this application, data selection can be performed on the operator execution data of each collective communication operator in a round of training steps, selecting some representative data to form a small data set, and then analyzing the small data set. In a specific implementation, the present application can trim the above target sequence according to the full set of collective communication operators provided by the collective communication library. The trimmed sequence obtained by trimming needs to be able to cover the full set of collective communication operators provided in the collective communication library. In other words, the operation types of each collective communication operator included in the sequence obtained after trimming can cover the operation types of the full set of collective communication operators in the collective communication library. In this way, the execution time of each collective communication operator in the trimmed sequence can fully reflect the normal fluctuations and potential anomalies in the execution time of operators of various operation types in this round of training steps, and the operator execution data of each collective communication operator in the trimmed sequence in the training step can be combined to form a small data set; 2. Feature extraction. The above small data set can be subjected to in-depth feature engineering processing.The feature engineering processing in this application may refer to statistical calculation processing of preset values ​​based on the execution time of each collective communication operator in the trimming sequence in the training step. The preset values ​​here may include but are not limited to: moving average, moving standard deviation, deviation standard deviation, quartiles, interquartile range and other values. Among them, the moving average refers to the average value of the operator execution time of each collective communication operator in the training step in the trimming sequence (which can be determined based on the start execution time and end execution time of each collective communication operator). This average value can be used to reflect the duration trend of the trimming sequence; the moving standard deviation is used to measure the performance of each collective communication operator in the training step in the trimming sequence. The degree of dispersion of the operator execution time can be used to reveal volatility; the deviation from the standard deviation can refer to the degree of deviation of the operator execution time of a certain collective communication operator in the trimmed sequence relative to the average value of the overall operator execution time in the trimmed sequence; the quartiles can refer to the three values ​​that divide the collective communication operators in the trimmed sequence into four equal parts, which can be used to describe the distribution of the operator execution time of each collective communication operator in the trimmed sequence in the training step, especially the median of the three values, which can best reflect the central trend; the interquartile range can be used to indicate the degree of dispersion of the operator execution time of each collective communication operator in the trimmed sequence in the training step, which helps to identify individual extreme values.

[0127] In summary, for the first training step, the operator execution data of each collective communication operator in the target sequence in the first training step is preprocessed to obtain the specific implementation process of the step features of the first training step, which can at least include: first obtaining the full set of collective communication operators provided by the collective communication library, and performing sequence trimming processing on the target sequence according to the full set of collective communication operators to obtain a trimmed sequence corresponding to the target sequence, which will include any collective communication operator in the full set of collective communication operators; based on the execution time of each collective communication operator in the trimmed sequence in the first training step, the operator execution data of each collective communication operator in the trimmed sequence can be feature engineered to obtain the execution features corresponding to each collective communication operator. The feature engineering processing here can refer to the above description and will not be repeated here. The content obtained after the feature engineering processing of each collective communication operator can be used as the execution feature of the operator in the first training step. Through the above feature engineering processing, the features of each collective communication operator can be enriched, and in addition to its own operator execution data, it will also have corresponding execution features.

[0128] After obtaining the execution features of each collective communication operator in the trimmed sequence in the first training step, its operator execution data and corresponding execution features can be assembled (for example, spliced), thereby obtaining the operator assembly features of each collective communication operator in the trimmed sequence; thereafter, the operator assembly features of each collective communication operator in the trimmed sequence can be fused (for example, spliced), thereby obtaining the step features of the first training step. An operator detection model can be called based on the step features of the first training step; wherein, the operator detection model here can refer to any artificial intelligence model with classification and recognition functions. Since the random forest model has powerful capabilities and good robustness and generalization performance when processing complex nonlinear relationships, the present application can preferably use the random forest model as the operator detection model for abnormal operator detection.

[0129] Through the model parameters of the operator detection model, the step features of the first training step can be feature analyzed, and the abnormal collective communication operator with abnormal execution time (that is, the collective communication operator with abnormal operator execution time) is output.

[0130] In order to improve the accuracy and rationality of the output results of the operator detection model, the present application can pre-train the operator detection model to optimize the model parameters of the operator detection model so that the output results of the operator detection model can become more and more accurate. The training process of the operator detection model may at least include: obtaining sample data for training the operator detection model. This sample data may refer to the sample communication log of the sample training step. It should be understood that the present application may use a model training business that has been executed in a historical time period (a time period earlier than the training time of the current model training business) as a sample training business. Any round of training steps in this sample training business can be used as a sample training step. In this way, multiple sample training steps can be obtained. The communication log generated by each sample training step during the training process can be used as a sample communication log for training the operator detection model. Similar to the content included in the communication log of each round of training steps mentioned above, the sample communication log of each sample training step here will also include multiple set communication operators participating in the sample training step (for ease of distinction, they can be referred to as sample set communication operators), as well as the operator execution data of each sample set communication operator in the sample training step. Moreover, the operator execution data of each sample set communication operator in the sample training step will also include the execution time of the sample set communication operator in the sample training step.

[0131] After obtaining the sample communication log of the sample training step, the sample set communication operators participating in the sample training step can be sorted according to the execution time of these sample set communication operators in the sample training step (similarly, based on the execution time of each sample set communication operator in the sample training step, the operator execution time of each sample set communication operator in the sample training step can be calculated and counted. In this way, the sample set communication operators can be sorted in order from large to small or from small to large according to the operator execution time). The sorted sequence can be called a sample sequence; for this sample sequence, the present application can also preprocess it, and the preprocessed data set obtained can be called the step feature of the sample training step; based on the step feature of the sample training step, an operator detection model that has not been trained (can be called an initial operator detection model) can be called, and the model parameters of the initial operator detection model can be used to perform data analysis on the step feature of the sample training step, and output abnormal sample set communication operators with abnormal execution time. After obtaining the abnormal sample set communication operator of the sample training step, the sample set communication operator that actually has abnormalities in the sample training step can be obtained. These sample set communication operators that actually have abnormalities can be used as sample set communication operator labels corresponding to the sample training step. According to the difference between the sample set communication operator label and the abnormal sample set communication operator output by the model, the initial operator detection model can be trained, and finally an operator detection model that meets the model convergence conditions is obtained.

[0132] In the process of training the initial operator detection model based on the difference between the sample set communication operator label and the abnormal sample set communication operator output by the model, the difference can be calculated through methods such as cross-validation (for example, the error value between the sample set communication operator label and the abnormal sample set communication operator output by the model is calculated using the cross-entropy loss function, and this error value can be used as the difference between the two). Based on this difference, the performance of the operator detection model is evaluated to ensure its generalization ability on unknown data. Based on the evaluation results, the model parameters of the operator detection model can be continuously tuned to improve the accuracy and reliability of the operator detection model in detecting abnormal operators. Through training, the operator detection model will learn how to determine whether the execution time of any set communication operator in a training step is abnormal based on the input data (i.e., the step characteristics of a training step). After training is completed, the operator detection model can be used to detect in real time whether the execution time of a set communication operator in the training process of any model training business is abnormal. Once an abnormal operator is detected, the cause can be further analyzed and corresponding optimization measures can be taken to improve training efficiency and model performance.

[0133] Step S304: Find abnormal devices from computing devices that execute abnormal set communication operators.

[0134] In this application, after detecting the abnormal step in the model training business and the abnormal set communication operator in the abnormal step, since the specific abnormal step and a specific abnormal set communication operator in the abnormal step have been located, the search range of the abnormal device has been greatly narrowed. At this time, the root cause of the abnormality can be located based on these abnormal set communication operators to find the computing device with the abnormality. Specifically, each computing device that executes this abnormal set communication operator in the abnormal step can be obtained from N computing devices. These computing devices can all be used as candidate abnormal devices. Then, the specific abnormal device can be found from these candidate abnormal devices in a certain way.

[0135] In certain scenarios, a one-size-fits-all approach can be adopted: all computing devices executing the abnormal set communication operator can be designated as abnormal devices. These devices can then be isolated or replaced. While this may also isolate or replace normally functioning computing devices, it ensures smooth model training and improves model training efficiency. The specific abnormal device can then be identified from the replaced candidate abnormal devices and repaired.

[0136] In an embodiment of the present application, based on the communication logs of each round of training steps in the model training business, the abnormal training steps can be detected first, and then the abnormal collective communication operators can be detected from the training steps, and then the abnormal devices can be found from the computing devices that execute the abnormal collective communication operators. In this way, by narrowing the search scope step by step, the abnormal devices can be found efficiently and accurately without interrupting or restarting the model training business, so as to realize equipment operation and maintenance, ensure the smooth execution of the model training business, and improve the efficiency of model training. In addition, for network operation and maintenance, many abnormal phenomena in the process of executing the model training business are manifested as network problems. After using this solution, it can be accurately determined whether it is a network problem or a problem on the computing device side.

[0137] Furthermore, the above describes a method for finding abnormal devices based on an abnormal set communication operator. This method treats each computing device that executes this abnormal set communication operator as an abnormal device. Although it can directly and efficiently handle computing devices that do have abnormalities in a timely manner, it will also mishandle some computing devices that are operating normally. Based on this, in order to further improve the accuracy of finding abnormal devices, this application provides a method for finding abnormal devices based on an abnormal set communication operator. This method still utilizes the communication logs of each round of training steps mentioned above. The specific process of finding abnormal devices based on the abnormal set communication operator will be introduced in detail below with reference to the accompanying drawings.

[0138] See Figure 4 , Figure 4 This is a flow chart of a method for finding abnormal devices based on an abnormal set communication operator provided by an embodiment of the present application. This flow can correspond to the above-mentioned flow of finding abnormal devices from a computing device that executes an abnormal set communication operator. Figure 4 As shown, the process may include at least the following steps S401 to S405:

[0139] Step S401: multiple collective communication operators participating in the first training step are determined as step participating operators.

[0140] In this application, for the sake of distinction, the multiple collective communication operators participating in the first training step may be referred to as step participating operators.

[0141] Step S402: Create a topology map for the first training step based on at least one computing processor that executes each step-participating operator; the topology map is composed of multiple operator topology maps, one operator topology map corresponds to one step-participating operator, and one operator topology map includes at least one node, and one node in the operator topology map corresponds to a computing processor of the step-participating operator corresponding to the operator topology map.

[0142] In the present application, based on the above, after monitoring and collecting the execution data of each GPU participating in the execution of each collective communication operator in a round of training steps, since different GPUs may participate in the execution of the same collective communication operator at the same time, the present application can merge the execution data of different GPUs of a collective communication operator, thereby forming the operator execution data of the collective communication operator in a round of training steps. In other words, the operator execution data of a collective communication operator in a round of training steps is composed of the execution data of each GPU that executes the collective communication operator, then the operator execution data of a collective communication operator in a round of training steps can actually also include the information of each GPU that executes the collective communication operator in this training step, that is, the operator execution data of each collective communication operator in a round of training steps includes the information of at least one computing processor (GPU) that executes the collective communication operator in this training step (for example, the identifier or name of the computing processor). In this way, after obtaining the original communication log of the first training step, the volume of the communication log is very large, and the present application can perform detailed and systematic processing and analysis on it. Specifically, the present application can create a topology map for the first training step based on at least one computing processor participating in the execution of each step. The topology map can clearly display the dependency relationship, communication path and other information between each GPU and each collective communication operator.

[0143] The specific implementation process of creating a topology map for the first training step based on at least one computing processor that executes each step participating operator may at least include: counting the number of at least one computing processor that executes each step participating operator, and creating T nodes for each step participating operator according to the number of at least one computing processor that executes each step participating operator, where T is the number of at least one computing processor that executes the step participating operator (since the number of computing processors that participate in the execution of different step participating operators is not exactly the same, the total number of nodes corresponding to each step participating operator may also be different, that is, T may change with the change in the number of computing processors that participate in the execution of the step participating operator), and one of the T nodes corresponds to a computing processor that executes the step participating operator; for example, for a certain step participating operator, assuming that the computing processors that participate in the execution of the step participating operator include GPU0, GPU1 and GPU2, then a corresponding node may be created for GPU0, GPU1 and GPU2 respectively, so that 3 nodes can be created, and these 3 nodes can be used as nodes created for the step participating operator.

[0144] After creating the corresponding nodes for each step participating operator, you can establish connecting edges between the T nodes corresponding to each step participating operator, thereby obtaining the operator topology graph corresponding to each step participating operator; then, you can aggregate the operator topology graphs corresponding to each step participating operator in the first training step, thereby obtaining the initial topology graph of the first training step. To understand the specific process of creating the initial topology graph, please refer to Figure 5 , Figure 5 This is a schematic diagram of creating an initial topology diagram provided by an embodiment of the present application. Figure 5 The embodiment shown is an embodiment that is explained by taking the step participating operators of the first training step including step participating operator a, step participating operator b, step participating operator c, and step participating operator a (the collective communication operator a is executed twice before and after the first training step. For the sake of distinction, the step participating operator a executed for the second time may be referred to as step participating operator a2) as an example. It is assumed that the computing processors participating in the execution of step participating operator a in the first training step are GPU0, GPU1, and GPU2; the computing processors participating in the execution of step participating operator b in the first training step are GPU0, GPU2, and GPU3; the computing processors participating in the execution of step participating operator c in the first training step are GPU3, GPU4, and GPU5; the computing processors participating in the execution of step participating operator a2 in the first training step are GPU1, GPU2, and GPU3. Then, Figure 5As shown, for step participant operator a, a corresponding node 51 can be created for GPU0, a corresponding node 52 can be created for GPU1, and a corresponding node 53 can be created for GPU2; for step participant operator b, a corresponding node 511 can be created for GPU0, a corresponding node 522 can be created for GPU2, and a corresponding node 533 can be created for GPU3; for step participant operator c, a corresponding node 5111 can be created for GPU3, a corresponding node 5222 can be created for GPU4, and a corresponding node 5333 can be created for GPU5; for step participant operator a2, a corresponding node 51111 can be created for GPU1, a corresponding node 52222 can be created for GPU2, and a corresponding node 53333 can be created for GPU3.

[0145] Furthermore, after creating corresponding nodes for the participating computing processors corresponding to each step participating operator, the nodes corresponding to the step participating operator can be obtained. For example, for step participating operator a, nodes 51, 52, and 53 corresponding to the step participating operator can be obtained. After that, a connection edge can be established between the nodes corresponding to each step participating operator, thereby obtaining a topological graph of the step participating operator including the nodes and the connection edges, which can be called the operator topological graph of the operator. For example, Figure 5 As shown, between the nodes 51, 52 and 53 corresponding to the step participant operator a, a connection edge can be established between any two nodes, so that an operator topology graph (50a) including the nodes 51, 52 and 53 and three connection edges can be obtained; similarly, for the step participant operator b, the step participant operator c and the step participant operator a2, the corresponding operator topology graphs can be obtained in the same way, namely the operator topology graph (50b), the operator topology graph (50c) and the operator topology graph (50d). After obtaining the operator topology graphs corresponding to the step participant operator a, the step participant operator b, the step participant operator c and the step participant operator a2, the operator topology graphs of these step participant operators can be merged to form the initial topology graph (500H) corresponding to the first training step, as shown in FIG. Figure 5 As shown, the initial topology diagram (500H) includes the operator topology diagram corresponding to the operator participating in any step.

[0146] Furthermore, after constructing the initial topology graph corresponding to the first training step, the present application can determine the corresponding node weight for each node in the initial topology graph, thereby including a topology graph with node weights, and the topology graph can be used as the topology graph of the first training step. The specific process of determining the corresponding node weight for each node in the initial topology graph can at least include: Here, taking the determination of the corresponding node weight of any node as an example, any node in the initial topology graph can be determined as the target node, and the step participating operator corresponding to the target node can be determined as the target operator; further, the operator type to which the target operator belongs can be obtained, and the operator type can include an abnormal type and a non-abnormal type. The abnormal type can be used to characterize this operator as an abnormal set communication operator; while the non-abnormal operator is used to characterize the operator as a non-abnormal set communication operator. It should be understood that since the abnormal set communication operators in the first training step have been output through the operator detection model, the operator types to which these abnormal set communication operators belong can be determined as abnormal types, while the operator types to which other non-abnormal set communication operators belong can be determined as non-abnormal types. After obtaining the operator type to which the target operator belongs, the weight calculation method of the target node can be determined according to the operator type to which the target operator belongs; the weight calculation method here can include a first calculation method and a second calculation method, wherein the first calculation method can refer to a method of calculating the node weight using a first weight calculation function, and the second calculation method can refer to a method of calculating the node weight using a second weight calculation function. For ease of understanding, the first weight calculation function and the second weight calculation function will be introduced in detail below. Among them, the first weight calculation function can be shown as formula (2):

[0147] W(O i )=Wexec(0 i )+ Wres(O i )+Wabn(O i ) Formula (2)

[0148] Where, as shown in formula (2), W(O i ) can be used to characterize node O i Corresponding node weight; Wexec(O i ) can be used to characterize node O i The corresponding computing processor executes the node O in the training step (such as the first training step) i The execution time of the operator participating in the corresponding step is the execution time of the node O iThe degree of influence of the node weight of a step-participating operator should be understood as the execution time of the operator in the training step is an important factor for evaluating whether the step-participating operator is abnormal. Based on this, the execution time of the operator in the training step can be used as an evaluation parameter of the node weight of the corresponding node. The longer the operator execution time, the greater the influence of the operator on the overall training, and therefore the weight of the corresponding node should be greater. The calculation method for the degree of influence of the operator duration can be shown as formula (3):

[0149] Wexec(O i )=α1×T i Formula (3)

[0150] Among them, α1 as shown in formula (3) can be used to represent a custom scale factor, T i Can be used to characterize node O i The corresponding computing processor executes the node O in the training step (such as the first training step) i The execution time of the operator participating in the corresponding step.

[0151] As shown in formula (2), Wres( i ) can be used to characterize node O i The corresponding resource usage affects the node weight; the resource usage refers to the influence of node O i The corresponding computing processor executes the node O in the training step (such as the first training step). i The amount of computing resources (such as GPU resources, CPU resources, etc.) used by the corresponding step participating operator in the training step, that is, the node O i The corresponding step participates in the resource usage of the operator. The calculation method for the influence of resource usage on node weight can be shown as formula (4):

[0152] Wres(O i )=α2×R i Formula (4)

[0153] Among them, α2 as shown in formula (4) can be used to represent a custom scale factor, R i Can be used to characterize node O i The resource usage of the corresponding step participating operator in the training step.

[0154] As shown in formula (2), Wabn(O i ) can be used to characterize node O iThe degree of influence of the abnormality of the corresponding step participating operator on the node weight; for the collective communication operator detected as abnormal, this application can assign it a higher weight to reflect its overall impact on the model training business. The calculation method for the degree of influence of the abnormality of the step participating operator on the node weight can be shown as formula (5):

[0155] Wabn(O i )=α3×A i Formula (5)

[0156] Among them, α3 as shown in formula (5) can be used to represent a custom scale factor, A i Can be used to characterize node O i Whether the corresponding step participating operator is abnormal, at node O i When the corresponding step participating operator is an abnormal operator, the A i The possible value is 1.

[0157] The second weight calculation function can be shown as formula (6):

[0158] W(O i )=Wexec(0 i )+ Wres(O i ) Formula (6)

[0159] Where, as shown in formula (6), W(O i ) can be used to characterize node O i Corresponding node weight; Wexec(O i ) and Wres(O i The meaning represented by ) can be found in the relevant description in the above formula (2), which will not be repeated here. It can be seen from formula (2) and formula (6) that for the operator detected as abnormal in this application, the node weight corresponding to each node will have an abnormal weight value.

[0160] Based on this, it can be seen that the specific process of determining the weight calculation method of the target node according to the abnormal type of the target operator can at least include: it can be determined whether the operator type of the target operator is an abnormal type. If the operator type of the target operator is an abnormal type, the weight calculation method of the target node can be determined as the first calculation method. It can be seen from the above formula (2) that the first calculation method is used to calculate the execution time, resource occupancy and abnormality degree (that is, the degree of influence of the abnormal situation on the node weight) of the target operator corresponding to the target node; and if the operator type of the target operator is a non-abnormal type, the weight calculation method of the target node can be determined as the second calculation method. It can be seen from the above formula (6) that the second calculation method is used to calculate the execution time and resource occupancy of the target operator without considering the abnormal situation of the target operator. After determining the weight calculation method of the target node, the weight calculation method of the target node can be used to calculate the node weight corresponding to the target node.

[0161] Step S403: Obtain a probability transfer matrix corresponding to the topology graph; the probability transfer matrix includes the probability that any two nodes in the topology graph transfer to each other.

[0162] In the present application, after calculating the node weights corresponding to each node in the topological graph, the corresponding probability transfer matrix can be determined for the topological graph based on the node weights corresponding to each node. This probability transfer matrix will include the probability of any two nodes in the topological graph transferring to each other. That is to say, based on the node weights corresponding to each node, the probability of any two nodes transferring to each other can be calculated according to the statistical algorithm of probability transfer. In this way, a probability transfer matrix can be formed.

[0163] Step S404 : moving from the first node to the second node in the topology graph according to the probability transfer matrix, and obtaining the frequency of each node in the topology graph being visited during the movement.

[0164] In this application, two different nodes can be customized in the topology map as the first node and the second node respectively, wherein the first node can be used as the starting node and the second node can be used as the ending node. Then, a simulated walk can be performed in the topology map to move from the first node to the second node. During this movement, different nodes in the topology map will be visited, and we can count the frequency of each node being visited during this movement. Subsequently, abnormal devices can be found based on the frequency of each node being visited during the movement.

[0165] It should be noted that in order to improve the reliability of the frequency of each node being accessed, the present application can perform multiple simulated walks in the topology map. Specifically, a random walk algorithm can be used to apply the random walk algorithm to the topology map. In this way, according to the probability transfer matrix, multiple random simulated walks can be performed in the topology map. Each random simulated walk moves from one node in the topology map to another node. When each random simulated walk is completed, the frequency of each node in the topology map being visited can be counted. After multiple random walks are completed, the total frequency of each node in the topology map being visited during multiple movements can be statistically calculated. Subsequently, abnormal devices can be found based on the total frequency of each node being visited during multiple movements.

[0166] Step S405 , searching for abnormal devices from the N computing devices according to the frequency at which each node in the topology graph is accessed during the movement process.

[0167] In this application, the specific process of finding abnormal devices from N computing devices based on the frequency at which each node in the topology map is accessed during movement can at least include: obtaining the maximum frequency among the frequencies at which each node in the topology map is accessed during movement, the node corresponding to this maximum frequency is most likely the root cause of the abnormality, determining the node indicated by the maximum frequency as the abnormal node, and determining the computing processor corresponding to the abnormal node as the abnormal processor, and then determining the computing device running the abnormal processor among the N computing devices as the abnormal device. The robustness of abnormal device search can be further improved through multiple random walks and weighted analysis.

[0168] In the embodiments of the present application, after successfully screening out the abnormal set of communication operators, it is particularly important to accurately locate the root cause of the abnormality. To this end, the present application innovatively proposes the concept of an operator topology graph, which aims to effectively locate the root cause of the abnormality through graph analysis technology. In order to overcome the real-time problem brought about by full data analysis, the present application can adopt an incremental data processing method. With each new round of training steps, its communication log can be collected and the newly added communication log can be analyzed and processed in real time, thereby reducing the computational load. By regularly updating and maintaining the operator topology graph, the timeliness and accuracy of the analysis results can be ensured; through graph traversal and analysis, the relationship between the abnormal operator and its upstream and downstream operators can be explored and the root cause of the abnormality (abnormal device) can be finally located. In addition, when constructing the operator topology graph, the present application not only needs to record the dependency relationship between the operators, but also defines the weight by considering the execution time, resource usage and whether the operator is an abnormal operator through the weighted method of the graph. This can effectively identify which operators play a key role in the abnormal process. For operators with heavy dependencies, their abnormalities may have a greater impact on the overall training. The topology map is updated incrementally and automatically during each training to ensure that the dependencies between operators reflect changes in the current training status at any time.

[0169] In general, statistical analysis of existing cases shows that abnormal problems in model training services are usually caused by problems in certain computing devices, such as GPU overheating, network card jitter, and excessive CPU load. Therefore, the focus of this application is to monitor the collective communication process of model training services, obtain detailed communication logs, and analyze the communication logs to ultimately locate the abnormal problem on the abnormal device. This will facilitate subsequent operation and maintenance personnel to further isolate and replace related equipment to improve model training efficiency.

[0170] Further, see Figure 6 , Figure 6 This is a structural diagram of a processing device for a model training service provided in an embodiment of the present application. The processing device for the model training service may be a computer program (including program code) running on a computer device, for example, the processing device for the model training service is an application software; the processing device for the model training service may be used to execute Figure 3 As shown in the method. Figure 6 As shown, the processing device 1 for the model training service may include: a log acquisition module 11, a step detection module 12, an operator determination module 13 and a device search module 14.

[0171] The log acquisition module 11 is used to obtain the business logs generated by the model training business; the business logs include the communication logs generated by each round of training steps. The communication logs of a round of training steps include multiple collective communication operators participating in the training steps, and the operator execution data of each collective communication operator in the training step;

[0172] a step detection module 12, configured to detect the first training step based on operator execution data of the plurality of collective communication operators participating in the first training step in the first training step, and obtain a detection result of the first training step;

[0173] an operator determination module 13 for determining an abnormal collective communication operator from the plurality of collective communication operators participating in the first training step based on operator execution data of the plurality of collective communication operators participating in the first training step in the first training step if the detection result of the first training step indicates that the first training step is an abnormal step;

[0174] The device search module 14 is configured to search for abnormal devices from computing devices executing abnormal set communication operators.

[0175] The specific implementation of the log acquisition module 11, the step detection module 12, the operator determination module 13 and the device search module 14 can be found in the above Figure 3 The description of steps S301 to S304 in the corresponding embodiment will not be repeated here.

[0176] In one embodiment, the operator execution data of each collective communication operator in a round of training steps includes the execution time of the collective communication operator in the training step;

[0177] The step detection module 12 detects the first training step based on the operator execution data of the multiple collective communication operators participating in the first training step in the first training step, and obtains the specific implementation method of the detection result of the first training step, including:

[0178] Among the multiple collective communication operators participating in the first training step, the step duration of the first training step is calculated based on the execution time of each collective communication operator in the first training step;

[0179] Obtain the duration of the second training step; the second training step refers to the training step other than the first training step in the W rounds of training steps;

[0180] Sort the duration of the first training step and the duration of the second training step in order of size to obtain a duration sequence corresponding to W rounds of training steps;

[0181] Based on the duration sequence and the step duration of the first training step, the first training step is detected to obtain a detection result of the first training step.

[0182] In one embodiment, the step detection module 12 detects the first training step based on the duration sequence and the step duration of the first training step, and obtains the detection result of the first training step in a specific implementation method, including:

[0183] The probability distribution fitting function is used to perform probability distribution fitting on the time series to obtain the probability distribution corresponding to the time series;

[0184] Solving the probability distribution according to a preset quantile to obtain a solution result of the probability distribution for the preset quantile;

[0185] If the duration of the first training step is longer than the solution result, the detection result of the first training step is determined to be a failed detection result; the failed detection result indicates that the first training step is an abnormal step;

[0186] If the step duration of the first training step is less than the solution result, the detection result of the first training step is determined to be a pass detection result; the pass detection result indicates that the first training step is a non-abnormal step.

[0187] In one embodiment, the operator execution data of each collective communication operator in a round of training steps includes the execution time of the collective communication operator in the training step;

[0188] The operator determination module 13 determines a specific implementation of an abnormal collective communication operator from the multiple collective communication operators participating in the first training step based on the operator execution data of the multiple collective communication operators participating in the first training step in the first training step, including:

[0189] sorting the plurality of collective communication operators participating in the first training step according to their respective execution times in the first training step to obtain a target sequence;

[0190] Preprocessing operator execution data of each collective communication operator in the target sequence in the first training step to obtain step features of the first training step;

[0191] Calling an operator detection model based on the step features of the first training step;

[0192] The step features of the first training step are analyzed using the model parameters of the operator detection model, and an abnormal set communication operator having abnormal execution time is output.

[0193] In one embodiment, the operator determination module 13 preprocesses the operator execution data of each collective communication operator in the target sequence in the first training step to obtain a specific implementation of the step feature of the first training step, including:

[0194] Obtain the full set of collective communication operators provided by the collective communication library, and perform sequence clipping on the target sequence according to the full set of collective communication operators to obtain a clipped sequence corresponding to the target sequence; the clipped sequence includes any collective communication operator in the full set of collective communication operators;

[0195] Based on the execution time of each collective communication operator in the trimmed sequence in the first training step, feature engineering processing is performed on the operator execution data of each collective communication operator in the trimmed sequence to obtain the execution features corresponding to each collective communication operator;

[0196] Assembling the operator execution data and corresponding execution features of each collective communication operator in the trimmed sequence in the first training step to obtain the operator assembly features of each collective communication operator in the trimmed sequence;

[0197] The operator assembly features of each collective communication operator in the clipping sequence are fused to obtain the step features of the first training step.

[0198] In one embodiment, the training process of the operator detection model includes:

[0199] Obtain a sample communication log for a sample training step; a sample training step refers to a round of training steps in a sample model training service. The sample communication log includes multiple sample set communication operators participating in the sample training step, as well as operator execution data of each sample set communication operator in the sample training step; the operator execution data of each sample set communication operator in the sample training step includes the execution time of the sample set communication operator in the sample training step;

[0200] Sort the multiple sample set communication operators by their execution times in the sample training steps to obtain a sample sequence;

[0201] Preprocessing the operator execution data of each sample set communication operator in the sample sequence to obtain the step features of the sample training step;

[0202] Based on the step features of the sample training steps, the initial operator detection model is called, and the step features of the sample training steps are analyzed using the model parameters of the initial operator detection model to output the communication operators of the abnormal sample set with abnormal execution time.

[0203] According to the difference between the communication operator labels of the sample set corresponding to the sample training step and the communication operator of the abnormal sample set, the initial operator detection model is trained to obtain the operator detection model.

[0204] In one embodiment, the operator execution data of each collective communication operator in a round of training steps includes at least one computation processor executing the collective communication operator in the training steps;

[0205] The specific implementation of the device search module 14 to find abnormal devices from the computing devices that execute the abnormal set communication operator includes:

[0206] Determine the plurality of collective communication operators participating in the first training step as step participating operators;

[0207] Creating a topology graph for a first training step based on at least one computational processor executing each step participant operator; the topology graph comprising a plurality of operator topology graphs, one operator topology graph corresponding to each step participant operator, one operator topology graph comprising at least one node, one node in the operator topology graph corresponding to a computational processor of the step participant operator corresponding to the operator topology graph;

[0208] Obtain the probability transfer matrix corresponding to the topology graph; the probability transfer matrix includes the probability that any two nodes in the topology graph transfer to each other;

[0209] Moving from a first node to a second node in the topology graph according to the probability transfer matrix, obtaining a frequency at which each node in the topology graph is visited during the movement;

[0210] According to the access frequency of each node in the topology graph during the movement, abnormal devices are found from N computing devices.

[0211] In one embodiment, the device search module 14 creates a topology map for the first training step based on at least one computing processor participating in the operator in each step, including:

[0212] Count the number of at least one computing processor executing each step participating operator, and create T nodes for each step participating operator according to the number of at least one computing processor executing each step participating operator; T is the number of at least one computing processor executing the step participating operator, and one node in the T nodes corresponds to a computing processor executing the step participating operator;

[0213] Establish connection edges between the T nodes corresponding to the participating operators in each step, and obtain the operator topology graph corresponding to the participating operators in each step;

[0214] Aggregate the operator topology graphs corresponding to the participating operators in each step of the first training step to obtain an initial topology graph of the first training step;

[0215] A corresponding node weight is determined for each node in the initial topological graph to obtain a topological graph of the first training step.

[0216] In one embodiment, the device search module 14 determines the corresponding node weight for each node in the initial topology graph in a specific implementation manner, including:

[0217] Determine any node in the initial topology graph as the target node, and determine the step participating operator corresponding to the target node as the target operator;

[0218] Determine the weight calculation method of the target node based on the operator type of the target operator;

[0219] The node weight corresponding to the target node is calculated using the target node's weight calculation method.

[0220] In one embodiment, the device search module 14 determines a specific implementation of a weight calculation method for a target node based on the anomaly type to which the target operator belongs, including:

[0221] If the operator type of the target operator is an abnormal type, the weight calculation method of the target node is determined to be the first calculation method; the first calculation method is used to calculate the execution time, resource usage and abnormality of the target operator;

[0222] If the operator type of the target operator is a non-exceptional type, the weight calculation method of the target node is determined to be the second calculation method; the second calculation method is used to calculate the execution time and resource usage of the target operator.

[0223] In one embodiment, the device search module 14 searches for abnormal devices from N computing devices based on the access frequency of each node in the topology map during the movement process. The specific implementation method includes:

[0224] Get the maximum frequency among the frequencies at which each node in the topology graph is visited during movement;

[0225] Determine the node indicated by the maximum frequency as an abnormal node;

[0226] The computing processor corresponding to the abnormal node is determined as the abnormal processor, and the computing device running the abnormal processor among the N computing devices is determined as the abnormal device.

[0227] In an embodiment of the present application, based on the communication logs of each round of training steps in the model training business, the abnormal training steps can be detected first, and then the abnormal collective communication operators can be detected from the training steps, and then the abnormal devices can be found from the computing devices that execute the abnormal collective communication operators. In this way, by narrowing the search scope step by step, the abnormal devices can be found efficiently and accurately without interrupting or restarting the model training business, so as to realize equipment operation and maintenance, ensure the smooth execution of the model training business, and improve the efficiency of model training. In addition, for network operation and maintenance, many abnormal phenomena in the process of executing the model training business are manifested as network problems. After using this solution, it can be accurately determined whether it is a network problem or a problem on the computing device side.

[0228] Further, see Figure 7 , Figure 7 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. Figure 7 As shown, the above-mentioned computer device 8000 may include: a processor 8001, a network interface 8004 and a memory 8005. In addition, the above-mentioned computer device 8000 also includes: a user interface 8003, and at least one communication bus 8002. The communication bus 8002 is used to realize the connection and communication between these components. The user interface 8003 may include a display screen (Display), a keyboard (Keyboard), and the user interface 8003 may optionally include a standard wired interface and a wireless interface. The network interface 8004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 8005 may be a high-speed RAM memory, or a non-volatile memory (non-volatile memory), such as at least one disk memory. The memory 8005 may optionally also be at least one storage device located away from the aforementioned processor 8001. As Figure 7 As shown, the memory 8005 as a computer-readable storage medium may include an operating system, a network communication module, a user interface module, and a device control application.

[0229] exist Figure 7 In the computer device 8000 shown, the network interface 8004 can provide network communication functions; the user interface 8003 is mainly used to provide an interface for user input; and the processor 8001 can be used to call the device control application stored in the memory 8005 to achieve:

[0230] Obtain the business logs generated by the model training business; the business logs include the communication logs generated by each round of training steps. The communication logs of a round of training steps include multiple collective communication operators participating in the training step, as well as the operator execution data of each collective communication operator in the training step;

[0231] Performing a detection on the first training step based on operator execution data of the plurality of collective communication operators participating in the first training step in the first training step to obtain a detection result of the first training step;

[0232] If the detection result of the first training step indicates that the first training step is an abnormal step, determining an abnormal collective communication operator from the plurality of collective communication operators participating in the first training step based on operator execution data of the plurality of collective communication operators participating in the first training step in the first training step;

[0233] Find abnormal devices from computing devices that execute abnormal set communication operators.

[0234] It should be understood that the computer device 8000 described in the embodiment of the present application can execute the above Figures 3 to 5 The description of the processing method of the model training business in the corresponding embodiment can also be performed as described above. Figure 6 The description of the processing device 1 for the model training service in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of adopting the same method will not be repeated here either.

[0235] In addition, it should be noted that: the embodiment of the present application also provides a computer-readable storage medium, and the computer-readable storage medium stores a computer program executed by the computer device 8000 for data processing mentioned above, and the computer program includes program instructions. When the processor executes the program instructions, the computer program can execute the above-mentioned data processing. Figures 3 to 5 The description of the processing method of the above-mentioned model training business in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of adopting the same method will not be repeated. For technical details not disclosed in the computer-readable storage medium embodiment involved in this application, please refer to the description of the method embodiment of this application.

[0236] The above-mentioned computer-readable storage medium can be a processing device for the model training service provided by any of the aforementioned embodiments or an internal storage unit of the above-mentioned computer device, such as a hard disk or memory of a computer device. The computer-readable storage medium can also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (smart media card, SMC), a secure digital (secure digital, SD) card, a flash card (flashcard), etc. equipped on the computer device. Furthermore, the computer-readable storage medium can also include both an internal storage unit of the computer device and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium can also be used to temporarily store data that has been output or is to be output.

[0237] In one aspect of the present application, a computer program product is provided, comprising a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the method provided in one aspect of the embodiments of the present application.

[0238] The terms "first", "second", etc. in the description, claims, and drawings of the embodiments of the present application are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "comprising" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, apparatus, product, or device comprising a series of steps or units is not limited to the listed steps or modules, but may optionally include steps or modules not listed, or may optionally include other step units inherent to these processes, methods, apparatuses, products, or devices.

[0239] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0240] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0241] The methods and related devices provided by the embodiments of the present application are described with reference to the method flow charts and / or structural diagrams provided by the embodiments of the present application. Specifically, each process and / or block in the method flow charts and / or structural diagrams, as well as the combination of processes and / or blocks in the flow charts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 Schematic diagram of one or more processes and / or structures Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including the instruction device, which implements the function specified in the process. Figure 1 Schematic diagram of one or more processes and / or structures Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the process. Figure 1 The flow or flows and / or structures illustrate the steps of the functions specified in one block or multiple blocks.

[0242] The above disclosure is only a preferred embodiment of the present application, and certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A method for processing a model training service, characterized in that: The method is applied to a model training architecture, which includes a collective communication library and N computing devices for executing a model training service; the model training service includes W rounds of training steps, and the N computing devices perform communication tasks in each round of training steps of the model training service through a collective communication operator provided by the collective communication library; N and W are both positive integers; the method is applied to a first training step, which refers to any round of training steps in the W rounds of training steps, and the method includes: Obtaining a business log generated by the model training business; the business log includes a communication log generated by each round of the training step, and the communication log of a round of the training step includes multiple collective communication operators participating in the training step, and operator execution data of each collective communication operator in the training step; Detecting the first training step based on operator execution data of a plurality of collective communication operators participating in the first training step in the first training step to obtain a detection result of the first training step; If the detection result of the first training step indicates that the first training step is an abnormal step, determining an abnormal collective communication operator from the multiple collective communication operators participating in the first training step based on operator execution data of the multiple collective communication operators participating in the first training step in the first training step; An abnormal device is found from computing devices that execute the abnormal set communication operator.

2. The method according to claim 1, characterized in that The operator execution data of each of the collective communication operators in one round of the training step includes the execution time of the collective communication operator in the training step; The detecting the first training step based on operator execution data of the plurality of collective communication operators participating in the first training step in the first training step to obtain a detection result of the first training step includes: Among the multiple collective communication operators participating in the first training step, calculating the step duration of the first training step by counting the execution time of each of the collective communication operators in the first training step; Obtaining a duration of a second training step; the second training step refers to a training step in the W rounds of training steps excluding the first training step; Sort the duration of the first training step and the duration of the second training step in order of size to obtain a duration sequence corresponding to the W rounds of training steps; Based on the duration sequence and the step duration of the first training step, the first training step is detected to obtain a detection result of the first training step.

3. The method according to claim 2, characterized in that The detecting the first training step based on the duration sequence and the step duration of the first training step to obtain the detection result of the first training step includes: Performing probability distribution fitting processing on the time sequence using a probability distribution fitting function to obtain a probability distribution corresponding to the time sequence; Solving the probability distribution according to a preset quantile to obtain a solution result of the probability distribution for the preset quantile; If the duration of the first training step is longer than the solution result, determining that the detection result of the first training step is a detection failure result; the detection failure result indicates that the first training step is an abnormal step; If the step duration of the first training step is less than the solution result, the detection result of the first training step is determined to be a pass detection result; the pass detection result indicates that the first training step is a non-abnormal step.

4. The method according to claim 1, wherein The operator execution data of each of the collective communication operators in one round of the training step includes the execution time of the collective communication operator in the training step; The determining, based on operator execution data of the plurality of collective communication operators participating in the first training step in the first training step, an abnormal collective communication operator from the plurality of collective communication operators participating in the first training step, comprises: sorting the plurality of collective communication operators participating in the first training step according to their respective execution times in the first training step to obtain a target sequence; preprocessing operator execution data of each collective communication operator in the target sequence in the first training step to obtain a step feature of the first training step; Calling an operator detection model based on the step features of the first training step; Feature analysis is performed on the step features of the first training step using the model parameters of the operator detection model, and an abnormal set communication operator having abnormal execution time is output.

5. The method according to claim 4, characterized in that The preprocessing of operator execution data of each collective communication operator in the target sequence in the first training step to obtain a step feature of the first training step includes: Obtaining a full set of collective communication operators provided by the collective communication library, and performing sequence clipping processing on the target sequence according to the full set of collective communication operators to obtain a clipped sequence corresponding to the target sequence; the clipped sequence includes any collective communication operator in the full set of collective communication operators; Based on the execution time of each collective communication operator in the trimmed sequence in the first training step, feature engineering processing is performed on the operator execution data of each collective communication operator in the trimmed sequence to obtain an execution feature corresponding to each collective communication operator; Assembling the operator execution data and corresponding execution features of each of the collective communication operators in the trimmed sequence in the first training step to obtain an operator assembly feature of each of the collective communication operators in the trimmed sequence; The operator assembly features of each collective communication operator in the clipping sequence are fused to obtain the step features of the first training step.

6. The method according to claim 4, characterized in that The training process of the operator detection model includes: Obtain a sample communication log of a sample training step; the sample training step refers to a round of training steps in a sample model training service, the sample communication log includes multiple sample set communication operators participating in the sample training step, and operator execution data of each of the sample set communication operators in the sample training step; the operator execution data of each of the sample set communication operators in the sample training step includes the execution time of the sample set communication operator in the sample training step; Sort the multiple sample set communication operators by their respective execution times in the sample training step to obtain a sample sequence; Preprocessing operator execution data of each sample set communication operator in the sample sequence to obtain step features of the sample training step; Calling an initial operator detection model based on the step features of the sample training step, performing data analysis on the step features of the sample training step using model parameters of the initial operator detection model, and outputting an abnormal sample set communication operator with abnormal execution time; According to the difference between the sample set communication operator label corresponding to the sample training step and the abnormal sample set communication operator, the initial operator detection model is trained to obtain the operator detection model.

7. The method according to claim 1, characterized in that operator execution data of each of the collective communication operators in a round of the training step, including at least one computation processor executing the collective communication operator in the training step; The step of finding an abnormal device from computing devices executing the abnormal set communication operator includes: Determining the plurality of collective communication operators participating in the first training step as step participating operators; Creating a topology graph for the first training step based on at least one computing processor that executes each participating operator in the step; the topology graph is composed of a plurality of operator topology graphs, one operator topology graph corresponds to one participating operator in the step, one operator topology graph includes at least one node, and one of the nodes in the operator topology graph corresponds to a computing processor of the participating operator in the step corresponding to the operator topology graph; Obtaining a probability transfer matrix corresponding to the topology graph; the probability transfer matrix includes the probability that any two nodes in the topology graph transfer to each other; Moving from a first node to a second node in the topological graph according to the probability transfer matrix, obtaining a frequency at which each node in the topological graph is visited during the movement; According to the frequency at which each node in the topology diagram is accessed during the movement process, an abnormal device is found from the N computing devices.

8. The method according to claim 7, characterized in that The step of creating a topology map for the first training step according to at least one computing processor participating in the operator executing each step comprises: Counting the number of at least one computing processor executing each participating operator in the step, and creating T nodes for each participating operator in the step according to the number of at least one computing processor executing each participating operator in the step; T is the number of at least one computing processor executing the participating operator in the step, and one node in the T nodes corresponds to a computing processor executing the participating operator in the step; Establishing connection edges between T nodes corresponding to the participating operators in each step to obtain an operator topology graph corresponding to each participating operator in the step; Aggregating the operator topology graphs corresponding to the participating operators in each step of the first training step to obtain an initial topology graph of the first training step; A corresponding node weight is determined for each node in the initial topological graph to obtain a topological graph of the first training step.

9. The method according to claim 8, characterized in that Determining a corresponding node weight for each node in the initial topology graph includes: Determine any node in the initial topology graph as a target node, and determine the step participating operator corresponding to the target node as a target operator; Determine a weight calculation method for the target node according to the operator type to which the target operator belongs; The node weight corresponding to the target node is calculated using the target node weight calculation method.

10. The method according to claim 9, characterized in that The determining of a weight calculation method of the target node according to the anomaly type to which the target operator belongs includes: If the operator type of the target operator is an abnormal type, the weight calculation method of the target node is determined to be a first calculation method; the first calculation method is used to calculate the execution time, resource usage and abnormality of the target operator; If the operator type of the target operator is a non-abnormal type, the weight calculation method of the target node is determined to be a second calculation method; the second calculation method is used to calculate the execution time and resource occupancy of the target operator.

11. The method according to claim 7, characterized in that The step of finding an abnormal device from the N computing devices according to a frequency at which each node in the topology graph is accessed during movement includes: Obtaining a maximum frequency among the frequencies at which each node in the topology graph is visited during movement; determining the node indicated by the maximum frequency as an abnormal node; The computing processor corresponding to the abnormal node is determined as an abnormal processor, and the computing device running the abnormal processor among the N computing devices is determined as an abnormal device.

12. A processing device for model training business, characterized in that: The device is applied to a model training architecture, which includes a collective communication library and N computing devices for executing a model training service; the model training service includes W rounds of training steps, and the N computing devices perform communication tasks in each round of training steps of the model training service through a collective communication operator provided by the collective communication library; N and W are both positive integers; the device is applied to a first training step, where the first training step refers to any one of the W rounds of training steps, and the device includes: A log acquisition module is used to obtain the business log generated by the model training business; the business log includes the communication log generated by each round of the training step, and the communication log of a round of the training step includes multiple collective communication operators participating in the training step, and operator execution data of each collective communication operator in the training step; a step detection module, configured to detect the first training step based on operator execution data of a plurality of collective communication operators participating in the first training step in the first training step, and obtain a detection result of the first training step; an operator determination module, configured to, if the detection result of the first training step indicates that the first training step is an abnormal step, determine an abnormal collective communication operator from the multiple collective communication operators participating in the first training step based on operator execution data of the multiple collective communication operators participating in the first training step in the first training step; The device search module is used to search for abnormal devices from computing devices that execute the abnormal set communication operator.

13. A computer device, characterized in that: include: processor, memory, and network interface; The processor is connected to the memory and the network interface, wherein the network interface is used to provide a network communication function, the memory is used to store a computer program, and the processor is used to call the computer program so that the computer device executes the method according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor and executing the method according to any one of claims 1 to 11.

15. A computer program product, characterized in that The computer program product comprises a computer program stored in a computer-readable storage medium. The computer program is suitable for being read and executed by a processor, so as to enable a computer device having the processor to perform the method according to any one of claims 1 to 11.

Images