Performance root cause analysis method and device for mixed expert model

Abstract

The specification provides a performance root cause analysis method and device of a mixed expert model; wherein the mixed expert model is distributed and deployed on a plurality of computing nodes; any computing node in the plurality of computing nodes is determined as a master node; the method is applied to the master node, comprising: obtaining communication waiting time consumption information periodically sent by each computing node; wherein the communication waiting time consumption information comprises communication waiting time consumption of each computing node waiting to receive tokens from each computing node; gathering the communication waiting time consumption information sent by each computing node into a communication waiting time consumption information set; performing outlier analysis on the communication waiting time consumption information set to determine abnormal communication waiting time consumption from the communication waiting time consumption information set; and based on the abnormal communication waiting time consumption, locating a performance abnormal root cause of the mixed expert model.

Description

Technical Field

[0001] This specification relates to one or more embodiments in the field of artificial intelligence, and more particularly to a method and apparatus for performance root cause analysis using a hybrid expert model. Background Technology

[0002] With the continuous growth in the scale of large language models (LLMs), the Mixture of Experts (MoE) architecture has been widely used in industrial-grade large model systems due to its good balance between model capacity and computational efficiency. The MoE architecture distributes input tokens (the basic units used by the model when processing data, usually called lexical units) to multiple sparsely activated subnetworks (i.e., "experts") through a dynamic routing mechanism, thereby significantly improving the model's expressive power without significantly increasing computational cost. Specifically, the MoE architecture consists of a gating network and a set of parallel expert subnetworks. For each input token, the gating network calculates its matching degree with each expert (usually represented by softmax-normalized weights) and selects only the Top K highest-scoring experts for forward propagation (K is usually 1 or 2). Since only a few experts are activated during each inference or training, the overall computational cost only increases linearly and slowly with the total number of experts, rather than exponentially, thus achieving the design goal of high capacity and low computation. This sparse activation mechanism allows the MoE architecture to build models with a large number of parameters while maintaining acceptable hardware resource consumption.

[0003] For models built on the MoE architecture, in their distributed deployment, experts are typically assigned to different devices (e.g., GPUs or compute nodes), with each device hosting a subset of experts and performing local computations. Since each input token is usually processed by only a few experts, and these selected experts are likely distributed across different devices, communication between devices is necessary to exchange data. Specifically, when a gating network on a device decides to route a token to a non-local expert, that device needs to send the token's representation to the device actually hosting that expert; correspondingly, after completing the expert's computation, that device also needs to use the result for subsequent aggregation. This cross-device data exchange is crucial for MoE models to achieve sparse activation and the collaborative operation of the global model. Because MoE models rely on complex cross-device communication during distributed inference or training, communication failures or performance anomalies can lead to training interruptions or inference errors.

[0004] Because the MoE model introduces features such as dynamic routing, sparse activation, and strong cross-device communication dependencies while increasing model capacity, its performance bottleneck is no longer limited to a single dimension. Therefore, root cause analysis of MoE model performance is essential and helps ensure and improve its performance. Summary of the Invention

[0005] In view of the above, one or more embodiments of this specification provide the following technical solutions: According to a first aspect of one or more embodiments of this specification, a performance root cause analysis method for a hybrid expert model is proposed; wherein the hybrid expert model is distributed and deployed on a plurality of computing nodes; any one of the plurality of computing nodes is determined as a master node; the method is applied to the master node, including: Obtain communication waiting time information periodically sent by each computing node; wherein, the communication waiting time information includes the communication waiting time for the computing node to wait to receive tokens from each computing node; The communication wait time information sent by each computing node is aggregated into a communication wait time information set; Outlier analysis is performed on the communication wait time information set to identify abnormal communication wait times from the communication wait time information set; Based on the abnormal communication waiting time, the root cause of the performance anomaly of the hybrid expert model is located.

[0006] According to a second aspect of one or more embodiments of this specification, a performance root cause analysis apparatus for a hybrid expert model is provided; wherein the hybrid expert model is distributed across a plurality of computing nodes; any one of the plurality of computing nodes is determined as a master node; the apparatus is applied to the master node, comprising: The acquisition module acquires communication waiting time information periodically sent by each computing node; wherein, the communication waiting time information includes the communication waiting time for the computing node to wait to receive tokens from each computing node; The aggregation module aggregates the communication wait time information sent by each computing node into a communication wait time information set; The analysis module performs outlier analysis on the communication waiting time information set to identify abnormal communication waiting times from the communication waiting time information set; The localization module locates the root cause of the performance anomaly of the hybrid expert model based on the abnormal communication waiting time.

[0007] According to a third aspect of one or more embodiments of this specification, an electronic device is provided, comprising: processor; Memory used to store processor-executable instructions; The processor executes the executable instructions to implement the steps of the method as described in any of the above embodiments.

[0008] According to a fourth aspect of one or more embodiments of this specification, a computer-readable storage medium is provided that stores computer instructions thereon, which, when executed by a processor, implement the steps of the method as described in any of the above embodiments.

[0009] According to a fifth aspect of one or more embodiments of this specification, a computer program product is provided, comprising a computer program / instructions that, when executed by a processor, implement the steps of the method as described in any of the above embodiments.

[0010] In the above technical solution, for the MoE model distributed across several computing nodes, the master node among these computing nodes can obtain the communication waiting time information periodically sent by each computing node. This communication waiting time information can include the communication waiting time for the computing node to receive tokens from each computing node. The communication waiting time information sent by each computing node is aggregated into a communication waiting time information set. Outlier analysis can then be performed on this communication waiting time information set to identify abnormal communication waiting times. Based on the identified abnormal communication waiting times, the root cause of the performance anomaly of the hybrid expert model can be located.

[0011] By employing the above approach, the root cause of the "Slow Problem" can be accurately and quickly located during the inference or training process of the MoE model, thereby ensuring the stability, predictability, and service quality of the MoE model's online service. Furthermore, by clearly separating the two core functions of communication wait time detection and outlier analysis, and utilizing an independent thread running on the master node to coordinate the entire process, it is possible to reuse the outlier analysis logic and achieve decentralized root cause analysis of MoE model performance, avoiding impacts on the online MoE model service. Attached Figure Description

[0012] Figure 1 This is a schematic diagram of a MoE model provided in an exemplary embodiment.

[0013] Figure 2 This is a flowchart of a performance root cause analysis method for a MoE model provided in an exemplary embodiment.

[0014] Figure 3 This is a schematic diagram of a communication waiting time matrix provided in an exemplary embodiment.

[0015] Figure 4 This is a schematic diagram of another communication wait time matrix provided in an exemplary embodiment.

[0016] Figure 5 This is a schematic diagram of the structure of a device provided in an exemplary embodiment.

[0017] Figure 6 This is a block diagram of a performance root cause analysis apparatus using a hybrid expert model, provided in an exemplary embodiment. Detailed Implementation

[0018] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0019] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this manual are all information and data authorized by the user or fully authorized by all parties. The collection, use and processing of related data shall comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals shall be provided for users to choose to authorize or refuse.

[0020] In the distributed deployment of the MoE model, experts are typically assigned to different devices (e.g., GPUs or compute nodes), with each device hosting a subset of experts and performing local computations. Since each input token is usually processed by only a few experts, and these selected experts are likely distributed across different devices, communication and data exchange between devices are necessary. Specifically, when a gating network on a device decides to route a token to a non-local expert, that device needs to send a representation of the token to the device actually hosting that expert; correspondingly, after completing the expert's computation, that device also needs to use the result for subsequent aggregation. This cross-device data exchange is crucial for the MoE model to achieve sparse activation and the collaborative operation of the global model capabilities.

[0021] To coordinate the computation and communication between devices used to deploy the MoE model, each device participating in inference can be assigned a unique rank (e.g., a process ID in the MPI or NCCL communication domain). This rank not only identifies the device's logical location in the network but also determines the subset of experts it hosts and its role during the communication phase (sender, receiver, or both). During MoE model inference, each rank first analyzes each input token in the current batch using its locally deployed gated network to determine which experts should handle that token. Since each rank typically hosts only a subset of all experts, for tokens assigned to non-local experts, the rank needs to send the feature data of those tokens to the other ranks that actually host the corresponding experts. Once all token routing decisions are complete, the system running the MoE model performs a global all-to-all communication operation, where each rank simultaneously sends the token data it needs to forward to all other possible ranks and receives tokens sent to it from other ranks. Because different inputs activate vastly different expert combinations, the amount of data that each rank needs to receive or send varies significantly across different batches. Therefore, the communication behavior and computational load of each rank are highly dependent on the distribution of experts activated by the current input, resulting in a dynamic and unbalanced inference process at the rank level.

[0022] This dependency leads to a highly unbalanced communication pattern in systems running the MoE model. Unbalanced communication refers to the significant differences in the amount of data received or sent by different ranks during the communication phase, under the expert parallel setting of the MoE model. Due to the semantic diversity of the input content, some ranks may be frequently selected as hosts for popular experts, thus continuously receiving a large number of cross-rank token requests in multiple batches; while experts hosted by other ranks may remain in a low-activity state for a long time, resulting in extremely low communication load. This drastic fluctuation in communication load not only causes some ranks to become communication bottlenecks, but also leads to problems such as uneven GPU utilization and prolonged synchronization waiting time. In addition, because the local computation completion time of each rank is highly asynchronous due to the number of active experts, the communication start time is also misaligned accordingly, making it difficult to efficiently align the entire inference pipeline, thus restricting system throughput and latency performance.

[0023] In unbalanced communication mode, the performance of a system running the MoE model is highly susceptible to slow nodes or slow links. Slow nodes refer to devices that are slow to respond during inference or training due to computational, memory, or scheduling bottlenecks; slow links refer to network connections with limited bandwidth, high latency, or packet loss in the communication path between devices. These anomalies directly slow down global synchronization operations (especially all-to-all communication), causing drastic fluctuations in key performance indicators such as Time To First Token (TTFT) and Time Per Output Token (TPOT), and in severe cases, even leading to request timeouts, service degradation, or task failures.

[0024] The causes of the above-mentioned "Slow problem" are complex and multi-sourced, and can usually be summarized into the following categories: (1) Uneven computational load: Due to the sparse activation characteristics of the MoE model, some experts may be frequently selected due to semantic preferences (i.e., "hot experts"), resulting in the rank hosting these experts bearing a much higher computational load than the average, forming a local computational bottleneck; (2) Network link anomalies: Including instantaneous failures in high-speed interconnection within devices, congestion or micro-burst packet loss in cross-device networks, which cause serious delays in data exchange between some ranks; (3) Host (main processor system, usually including CPU and its running operating system and drivers) side scheduling and resource competition: For example, CPU thread pool exhaustion, operating system scheduling delay, excessive GPU context switching overhead, or multiple processes / threads competing for the same communication resources, causing communication startup delay; (4) Defects in the implementation of communication libraries or frameworks: For example, deadlock, buffer overflow, retry mechanism failure, etc., in specific topologies or load modes of NCCL, MPI or custom communication primitives, which further amplify the negative impact of imbalance.

[0025] The "Slow Problem" that occurs during inference or training of the MoE model is essentially a system performance degradation phenomenon. Performance Root Cause Analysis (GRU) refers to the process of systematically identifying the root cause of performance degradation in a system, application, or service using a structured, logically clear, repeatable, and comprehensive approach. In practical applications, accurately and quickly identifying the root cause of the "Slow Problem" is crucial for ensuring the stability, predictability, and service quality of the MoE model's online service. Without in-depth GRU analysis, it is impossible to effectively distinguish between slow computation, slow communication, and slow scheduling, and therefore impossible to implement precise load balancing, expert reallocation, communication optimization, or fault isolation strategies.

[0026] This specification provides one or more embodiments of a technical solution for implementing root cause analysis of the performance of a MoE model, aiming to pinpoint the root cause of the "Slow problem" in an unbalanced communication mode of the MoE model. In this technical solution, for a MoE model distributed across several computing nodes, the master node among these computing nodes can obtain communication waiting time information periodically sent by each computing node. This communication waiting time information can include the communication waiting time for each computing node to receive tokens from other computing nodes. The communication waiting time information sent by each computing node is aggregated into a communication waiting time information set. Outlier analysis can then be performed on this communication waiting time information set to identify abnormal communication waiting times. Based on the identified abnormal communication waiting times, the root cause of the performance anomaly of the hybrid expert model can be located.

[0027] By employing the above approach, the root cause of the "Slow Problem" can be accurately and quickly located during the inference or training process of the MoE model, thereby ensuring the stability, predictability, and service quality of the MoE model's online service. Furthermore, by clearly separating the two core functions of communication wait time detection and outlier analysis, and utilizing an independent thread running on the master node to coordinate the entire process, it is possible to reuse the outlier analysis logic and achieve decentralized root cause analysis of MoE model performance, avoiding impacts on the online MoE model service.

[0028] The technical solution for implementing the MoE model is described in detail below.

[0029] refer to Figure 1 , Figure 1 This is a schematic diagram of a MoE model provided in an exemplary embodiment.

[0030] In practical applications, the MoE model can be deployed in a distributed manner. That is, the MoE model can be deployed on a distributed system, which may include several computing nodes (usually referring to independent computing units capable of communication). It should be noted that any one of these computing nodes can be designated as the master node, which executes the specific steps of the MoE model's performance root cause analysis. For example, an independent thread can be started on the master node to perform the MoE model's performance root cause analysis.

[0031] In the distributed deployment of the MoE model, experts can be assigned to different computing nodes, each responsible for hosting a subset of experts and performing local computations. To coordinate the computations and communication between the computing nodes used to deploy the MoE model, each computing node participating in inference can be assigned a unique rank. This rank not only identifies the device's logical location in the network but also determines the subset of experts it hosts and its role during communication. In this specification, the first rank (e.g., rank0) can be designated as the master node.

[0032] Combination Figure 1 ,refer to Figure 2 , Figure 2 This is a flowchart of a performance root cause analysis method for a MoE model provided in an exemplary embodiment.

[0033] The performance root cause analysis method of the above MoE model can be applied to, for example... Figure 1 The main node shown; the method may include the following steps: Step 202: Obtain the communication waiting time information periodically sent by each computing node; wherein, the communication waiting time information includes the communication waiting time for the computing node to wait to receive tokens from each computing node.

[0034] In this embodiment, for the computing nodes used for the distributed deployment of the above-mentioned MoE model, the communication waiting time information periodically sent by each computing node can be obtained first. Specifically, for any computing node, the communication waiting time information sent by the computing node can include the time spent by the computing node waiting to receive tokens (i.e., words) from each computing node (which can be referred to as communication waiting time).

[0035] For example, assuming that during one inference or training cycle of the MoE model described above, experts are activated on three computing nodes: rank0, rank1, and rank2, with rank0 being the master node, then rank0 can obtain communication wait time information periodically sent by rank0, rank1, and rank2. Specifically, the communication wait time information from rank0 can include: the communication wait time for rank0 to receive a token from rank1, and the communication wait time for rank0 to receive a token from rank2; the communication wait time information from rank1 can include: the communication wait time for rank1 to receive a token from rank0, and the communication wait time for rank1 to receive a token from rank2; and the communication wait time information from rank2 can include: the communication wait time for rank2 to receive a token from rank0, and the communication wait time for rank2 to receive a token from rank1.

[0036] In some embodiments, for each computing node, the communication waiting time for the computing node to receive a token from a certain computing node (which may be referred to as the target computing node) may be the average time for the computing node to wait to receive each token from the target computing node during one inference or training process of the MoE model; or, the communication waiting time for the computing node to wait to receive a token from the target computing node may be the average time for the computing node to wait to receive each token from the target computing node during the Dispatch (data distribution) phase of one inference or training process of the MoE model; or, the communication waiting time for the computing node to wait to receive a token from the target computing node may be the average time for the computing node to wait to receive each token (specifically, the processing result of the token) from the target computing node during the Combine (result merging) phase of one inference or training process of the MoE model.

[0037] Specifically, during a single inference or training iteration of the MoE model, Dispatch and Combine are two key operational steps used to perform conditional computation, which involves activating only a subset of experts to process the input, thereby expanding the model's capacity without significantly increasing computational cost. Dispatch aims to route the input tokens to the most suitable experts; Combine aims to weight and fuse the processing results of each expert on the tokens according to gating weights to obtain the final output. Therefore, for each computing node, the average time spent waiting to receive each token (or the processing result of the token) from each computing node can be calculated in each round of the Dispatch and Combine phase of the above MoE model. For example, assuming that the computing node in a certain round of the Dispatch and Combine phase of the MoE model has a duration of T from the start of receiving tokens from a certain target computing node to the end of receiving tokens from that target computing node, and a total of N tokens are received, then the average time spent by the computing node waiting to receive each token from the target computing node is T / N, that is, the time spent by the computing node waiting to receive tokens from the target computing node is T / N.

[0038] Alternatively, the Dispatch and Combine phases can be separated and their time consumption calculated separately. That is, for each computing node, the average time spent waiting for each token to be received from each computing node can be calculated separately in each round of the Dispatch phase of the MoE model, or the average time spent waiting for each token to be received from each computing node can be calculated separately in each round of the Combine phase of the MoE model. In this scenario, each computing node can utilize the calculated average time of the Dispatch phase to construct communication waiting time information for the Dispatch phase. This allows for outlier analysis of the communication waiting time information set aggregated from the Dispatch phase information from each computing node, thus enabling the identification of the root cause of performance anomalies in the MoE model during the Dispatch phase. Alternatively, each computing node can utilize the calculated average time of the Combine phase to construct communication waiting time information for the Combine phase. This allows for outlier analysis of the communication waiting time information set aggregated from the Combine phase information from each computing node, thus enabling the identification of the root cause of performance anomalies in the MoE model during the Combine phase.

[0039] It should be noted that, for each computing node, the communication waiting time information periodically sent by the computing node to the aforementioned master node, which includes the waiting time for receiving a token from a certain target computing node, can be the time most recently determined for the computing node to receive a token from the target computing node since the current sending time.

[0040] Step 204: Aggregate the communication wait time information sent by each computing node into a communication wait time information set.

[0041] In this embodiment, after obtaining the communication waiting time information sent by each computing node within the same period, this communication waiting time information can be further aggregated into a communication waiting time information set, so that outlier analysis can be performed on this communication waiting time information set to locate the root cause of the performance anomaly of the above-mentioned MoE model.

[0042] For example, a new set of communication wait time information can be created, and all communication wait times from these communication wait time information can be added to that set of communication wait time information.

[0043] In some embodiments, for the convenience of subsequent analysis, the above-mentioned communication waiting time information set can specifically be a set in matrix form; that is, when the above-mentioned communication waiting time information is obtained, a communication waiting time matrix can be constructed based on the communication waiting time in this communication waiting time information.

[0044] like Figure 3 As shown, for each element in the constructed communication wait time matrix, the communication wait time represented by the element can be the communication wait time for the computing node corresponding to the row of the element to receive the token from the computing node corresponding to the column of the element; for example, the element t01 in the first row and second column of the communication wait time matrix can be the communication wait time for rank0 to wait to receive the token from rank1.

[0045] Or, such as Figure 4 As shown, for each element in the constructed communication wait time matrix, the communication wait time represented by the element can be the communication wait time for the computing node corresponding to the column of the element to receive the token from the computing node corresponding to the row of the element; for example, the element t10 in the first row and second column of the communication wait time matrix can be the communication wait time for rank1 to wait to receive the token from rank0.

[0046] Step 206: Perform outlier analysis on the communication waiting time information set to identify abnormal communication waiting times from the communication waiting time information set.

[0047] In this embodiment, outlier analysis can be performed on the aforementioned communication wait time information set. Outlier analysis, also known as anomaly detection or outlier point detection, refers to the process of identifying anomalous data points (i.e., "outliers" or "outliers") in a dataset that significantly deviate from the majority of data patterns, behaviors, or distributions. Therefore, by performing outlier analysis on this communication wait time information set, abnormal communication wait times can be determined from it, thereby allowing the identification of the root cause of performance anomalies in the aforementioned MoE model based on the identified abnormal communication wait times.

[0048] In some embodiments, as described above, the communication wait time information set may specifically be a set in matrix form. In this case, outlier analysis can be performed on the rows, columns, and / or elements of the communication wait time matrix to identify anomalous communication wait times.

[0049] In some embodiments, on the one hand, for each row in the aforementioned communication waiting time matrix, when performing outlier analysis on that row, the average communication waiting time in that row can be calculated as the average communication waiting time corresponding to that row, and based on the average communication waiting time corresponding to that row, it can be determined whether that row belongs to an outlier row. For example... Figure 3Taking the communication waiting time matrix shown as an example, the average communication waiting time corresponding to the first row is: t(row0) = (t00 + t01 + t02) / 3. Therefore, based on the average communication waiting time corresponding to each row in the communication waiting time matrix, abnormal rows can be identified from the matrix.

[0050] On the other hand, for each column in the aforementioned communication waiting time matrix, when performing outlier analysis on that column, the average communication waiting time in that column can be calculated as the average communication waiting time corresponding to that column. Based on this average communication waiting time, it can be determined whether the column belongs to an outlier column. For example... Figure 3 Taking the communication waiting time matrix shown as an example, the average communication waiting time corresponding to the first column is: t(column0)=(t00+t10+t20) / 3. Therefore, based on the average communication waiting time corresponding to each column in the communication waiting time matrix, abnormal columns can be determined from the matrix.

[0051] Furthermore, when performing outlier analysis on each element in the aforementioned communication waiting time matrix, it is possible to determine whether the element is an anomalous element based on the communication waiting time it represents. Thus, anomalous communication waiting times can be identified from the communication waiting time matrix based on the communication waiting times represented by each element.

[0052] Furthermore, in some embodiments, when determining whether a row is an abnormal row based on the average communication waiting time corresponding to that row in the aforementioned communication waiting time matrix, it can be directly determined whether the average communication waiting time corresponding to that row is greater than a preset threshold. If the average communication waiting time corresponding to that row is greater than the threshold, then that row can be determined to be an abnormal row.

[0053] Alternatively, to improve the accuracy of outlier analysis, specifically the accuracy of identifying anomalous rows, we can first calculate the average communication wait time corresponding to each row in the aforementioned communication wait time matrix, and then calculate the standard deviation of the average communication wait time corresponding to each row in the matrix. Then, for each row in the communication wait time matrix, to determine whether a row is an anomalous row, we can specifically calculate the ratio of the difference between the average communication wait time corresponding to that row and the average value, to the standard deviation, and determine whether this ratio is greater than a preset threshold. It should be noted that, to minimize bias, a constant can be set, thereby adjusting this ratio to be the ratio of the difference between the average communication wait time corresponding to that row and the average value, to the difference between the standard deviation and the constant. If this ratio is greater than the threshold, then the row can be identified as an anomalous row.

[0054] For example Figure 3 Taking the communication wait time matrix shown as an example, the average communication wait time corresponding to each row in the matrix is: The standard deviation of the average communication wait time corresponding to each row in the communication wait time matrix is: For the first row of this communication wait time matrix, the ratio can be determined. (or , If the ratio (which is a constant) is greater than a preset threshold, then the first row in the communication wait time matrix can be determined to be an abnormal row.

[0055] In other embodiments, when determining whether a column is an abnormal column based on the average communication waiting time corresponding to that column in the aforementioned communication waiting time matrix, it can be directly determined whether the average communication waiting time corresponding to that column is greater than a preset threshold. If the average communication waiting time corresponding to that column is greater than the threshold, then that column can be determined to be an abnormal column.

[0056] Alternatively, to improve the accuracy of outlier analysis, specifically the accuracy of identifying outliers, we can first calculate the average communication wait time corresponding to each column in the aforementioned communication wait time matrix, and then calculate the standard deviation of the average communication wait time corresponding to each column in the same matrix. Then, for each column in the communication wait time matrix, to determine if a column is an outlier, we can specifically calculate the ratio of the difference between the average communication wait time corresponding to that column and the average value, to the standard deviation, and then determine if this ratio is greater than a preset threshold. It should be noted that, to minimize bias, a constant can be set, thereby adjusting this ratio to be the ratio of the difference between the average communication wait time corresponding to that column and the average value, to the difference between the standard deviation and the constant. If this ratio is greater than the threshold, then the column can be identified as an outlier.

[0057] For example Figure 3 Taking the communication waiting time matrix shown as an example, the average communication waiting time corresponding to the columns in the communication waiting time matrix is: The standard deviation of the average communication wait time corresponding to the columns in this communication wait time matrix is: For the first row of this communication wait time matrix, the ratio can be determined. (or , If the ratio (which is a constant) is greater than a preset threshold, then the first row in the communication wait time matrix can be determined to be an abnormal row.

[0058] In some other embodiments, when determining whether the communication waiting time represented by each element in the aforementioned communication waiting time matrix is ​​abnormal, it can be directly determined whether the communication waiting time represented by the element is greater than a preset threshold. If the communication waiting time represented by the element is greater than the threshold, then the element can be determined to be an abnormal communication waiting time.

[0059] Alternatively, to improve the accuracy of outlier analysis, specifically the accuracy of identifying anomalous communication wait times, we can first calculate the average communication wait time represented by each element in the aforementioned communication wait time matrix, and then calculate the standard deviation of the communication wait times represented by each element in the matrix. Then, for each element in the communication wait time matrix, to determine whether the communication wait time represented by that element is anomalous, we can specifically calculate the ratio of the difference between the communication wait time represented by that element and the average value, to the standard deviation, and determine whether this ratio is greater than a preset threshold. It should be noted that, to minimize bias, a constant can be set, thereby adjusting the ratio to be the ratio of the difference between the communication wait time represented by that element and the average value, to the difference between the standard deviation and the constant. If this ratio is greater than the threshold, then the communication wait time represented by that element can be determined to be anomalous.

[0060] For example Figure 3 Taking the communication wait time matrix shown as an example, the average communication wait time represented by the elements in this matrix is: The standard deviation of the communication wait time corresponding to the column in the communication wait time matrix is: For the element t01 in the first row and second column of the communication wait time matrix, the ratio can be determined. (or , If the ratio (which is a constant) is greater than a preset threshold, then the communication waiting time represented by element t01 in the communication waiting time matrix can be determined to be an abnormal communication waiting time.

[0061] Step 208: Based on the abnormal communication waiting time, locate the root cause of the performance anomaly of the hybrid expert model.

[0062] In this embodiment, after identifying abnormal communication waiting times from the aforementioned communication waiting time information set, the root cause of the performance anomaly of the MoE model can be located based on the identified abnormal communication waiting time.

[0063] In some embodiments, as described above, the communication waiting time information set can specifically be a set in matrix form. In this case, if there is an abnormal row in the communication waiting time matrix, it indicates that the computing node corresponding to the abnormal row is itself abnormal; if there is an abnormal column in the communication waiting time matrix, it indicates that the computing node corresponding to the abnormal column is itself abnormal; if there is an abnormal communication waiting time in the communication waiting time matrix, it indicates that the link between the computing node corresponding to the row of the abnormal communication waiting time and the computing node corresponding to the column of the abnormal communication waiting time is abnormal.

[0064] Specifically, in the aforementioned communication waiting time matrix, each element represents the time taken for the computing node corresponding to the row of that element to wait for the computing node corresponding to the column of that element to receive a token. If an abnormal row is identified in the communication waiting time matrix, it can be determined that the computing nodes corresponding to each abnormal row (in this case, these computing nodes are actually nodes receiving tokens) have data reception abnormalities, such as slow computation or congestion issues. If an abnormal column is identified in the communication waiting time matrix, it can be determined that the computing nodes corresponding to each abnormal column (in this case, these computing nodes are actually nodes sending tokens) have data transmission abnormalities, such as slow computation leading to transmission delays. If an abnormal communication waiting time is identified in the communication waiting time matrix, it can be determined that the link between the computing nodes corresponding to the rows of abnormal communication waiting times and the computing nodes corresponding to the columns of abnormal communication waiting times is abnormal.

[0065] In the aforementioned communication wait time matrix, each element represents the time taken for the computing node corresponding to the column of that element to wait for the computing node corresponding to the row of that element to receive a token. If an abnormal row is identified in the communication wait time matrix, it can be determined that the computing nodes corresponding to each abnormal row (in this case, these computing nodes are actually the nodes that send tokens) have data transmission abnormalities, such as slow computation leading to transmission delays. If an abnormal column is identified in the communication wait time matrix, it can be determined that the computing nodes corresponding to each abnormal column (in this case, these computing nodes are actually the nodes that receive tokens) have data reception abnormalities, such as slow computation or congestion. If an abnormal communication wait time is identified in the communication wait time matrix, it can be determined that the link between the computing nodes corresponding to the rows of abnormal communication wait times and the computing nodes corresponding to the columns of abnormal communication wait times is abnormal.

[0066] In practical applications, if a computing node experiences data transmission anomalies during the Dispatch phase, it usually indicates that the attention mechanism of that computing node is being computed unevenly. For example, the attention mechanism of that computing node may be taking longer to compute, resulting in late data transmission. Similarly, if a computing node experiences data transmission anomalies during the Combine phase, it usually indicates that the MoE expert computation of that computing node is unevenly distributed. For example, the MoE expert computation of that computing node may be taking longer to compute, resulting in late data transmission.

[0067] If a computing node experiences data reception anomalies during the Dispatch phase, it typically indicates uneven computation of its attention mechanism. For example, the attention mechanism might be computationally faster, leading to an earlier transition to the next state. Conversely, if a computing node experiences data transmission anomalies during the Combine phase, it usually indicates uneven computation of its MoE (MoE) experts. For instance, the MoE experts might be computationally faster, leading to an earlier transition to the next state. Furthermore, data reception anomalies could also indicate that the computing node is a hotspot expert, causing congestion at the receiving end.

[0068] If there is an anomaly in the link between two computing nodes, it usually indicates that the intra-machine communication link of one of the two computing nodes is abnormal, or the inter-machine communication link between the two computing nodes is abnormal.

[0069] In one or more embodiments of the present application, for a MoE model distributed across several computing nodes, the master node among these computing nodes can obtain the communication waiting time information periodically sent by each computing node. This communication waiting time information may include the communication waiting time for the computing node to receive tokens from each computing node. The communication waiting time information sent by each computing node is aggregated into a communication waiting time information set. Outlier analysis can be performed on this communication waiting time information set to identify abnormal communication waiting times. Based on the identified abnormal communication waiting times, the root cause of the performance anomaly of the hybrid expert model can be located.

[0070] By employing the above approach, the root cause of the "Slow Problem" can be accurately and quickly located during the inference or training process of the MoE model, thereby ensuring the stability, predictability, and service quality of the MoE model's online service. Furthermore, by clearly separating the two core functions of communication wait time detection and outlier analysis, and utilizing an independent thread running on the master node to coordinate the entire process, it is possible to reuse the outlier analysis logic and achieve decentralized root cause analysis of MoE model performance, avoiding impacts on the online MoE model service.

[0071] Corresponding to the aforementioned embodiments of the performance root cause analysis method of the hybrid expert model, this specification also provides embodiments of the performance root cause analysis apparatus of the hybrid expert model.

[0072] refer to Figure 5 , Figure 5 This is a schematic diagram of the structure of a device provided in an exemplary embodiment. For example... Figure 5 As shown, device 500 mainly consists of a communication interface 502, a user interface 504, a processor 506, and a data storage 508. These components are interconnected and communicate with each other via a system bus, network, or other connection mechanism 510. The communication interface 502 enables device 500 to communicate with other devices, access networks, and transmission networks via analog or digital modulation. For example, the communication interface 502 may include a chipset and antenna for wireless communication with a radio access network or access point. Furthermore, the communication interface 502 can be a wired interface such as Ethernet, Token Ring, or a USB port, or a wireless interface such as Wi-Fi, Bluetooth, Global Positioning System (GPS), or a wide-area wireless interface (e.g., WiMAX or LTE). Of course, the communication interface 502 can also support other forms of physical layer interfaces and standard or proprietary communication protocols. The communication interface 502 may also include multiple physical communication interfaces, such as Wi-Fi interfaces, Bluetooth interfaces, and wide-area wireless interfaces.

[0073] User interface 504 includes receiving user input and providing output to the user. Therefore, user interface 504 may include input components such as a keypad, keyboard, touch-sensitive or presence-sensitive panel, computer mouse, trackball, joystick, microphone, still camera, and video camera, and output components such as a display screen (which may be combined with a touch-sensitive panel), CRT, LCD, LED, display using DLP technology, printer, and other similar devices known or developed in the future. User interface 504 may also generate auditory output via speakers, speaker jacks, audio output ports, audio output devices, headphones, and other similar devices known or developed in the future. In some embodiments, user interface 504 may include software, circuitry, or other forms of logic capable of transmitting and receiving data from external user input / output devices. Additionally or alternatively, device 500 may support remote access from other devices via communication interface 502 or another physical interface (not shown). User interface 504 may be configured to receive user input, the position and movement of which may be indicated by indicators or cursors described herein. User interface 504 may also be configured as a display device for rendering or displaying text fragments.

[0074] Processor 506 may contain one or more general-purpose processors and / or special-purpose processors.

[0075] Data storage 508 may include one or more volatile and / or non-volatile storage components and may be integrated wholly or partially with processor 506. Data storage 508 may include removable and non-removable components.

[0076] Processor 506 is capable of executing program instructions 518 (e.g., compiled or uncompiled program logic and / or machine code) stored in data storage 508 to perform the various functions described herein. Data storage 508 may contain a non-transitory computer-readable medium on which program instructions are stored, which, when executed by device 500, enable device 500 to perform any methods, processes, or functions disclosed in this specification and / or the accompanying drawings. Execution of program instructions 518 by processor 506 may result in processor 506 using data 512.

[0077] For example, program instructions 518 may include an operating system 522 (e.g., an operating system kernel, device drivers, and / or other modules) installed on device 500 and one or more applications 520 (e.g., a browser, social application, or game application). Similarly, data 512 may include operating system data 516 and application data 514. Operating system data 516 is primarily accessible to the operating system 522, while application data 514 is primarily accessible to one or more applications 520. Application data 514 may reside in a file system visible or hidden from the user of device 500.

[0078] Application 520 can communicate with operating system 522 through one or more application programming interfaces (APIs). These APIs help application 520 read and / or write application data 514, transmit or receive information via communication interface 502, receive or display information on user interface 504, etc.

[0079] In some terminology, application 520 may be simply referred to as "app". Furthermore, application 520 can be downloaded to device 500 through one or more online app stores or app markets. However, applications can also be installed on device 500 in other ways, such as through a web browser or a physical interface on device 500 (e.g., a USB port).

[0080] refer to Figure 6 , Figure 6 This is a block diagram of a performance root cause analysis apparatus using a hybrid expert model, as provided in an exemplary embodiment. This hybrid expert model performance root cause analysis apparatus can be applied to, for example... Figure 5 The device shown is used to implement the technical solution of this specification. The hybrid expert model is distributed across several computing nodes; any one of the computing nodes is designated as the master node; the device is applied to the master node and includes: The acquisition module 602 acquires communication waiting time information periodically sent by each computing node; wherein, the communication waiting time information includes the communication waiting time for the computing node to wait to receive tokens from each computing node; The aggregation module 604 aggregates the communication waiting time information sent by each computing node into a communication waiting time information set; Analysis module 606 performs outlier analysis on the communication waiting time information set to identify abnormal communication waiting times from the communication waiting time information set; The positioning module 608 locates the root cause of the performance anomaly of the hybrid expert model based on the abnormal communication waiting time.

[0081] In some embodiments, the step of aggregating the communication wait time information sent by each computing node into a communication wait time information set includes: Based on the communication wait time information sent by each computing node, a communication wait time matrix is ​​constructed; wherein each element in the communication wait time matrix represents the communication wait time for the computing node corresponding to the row of the element to receive a word from the computing node corresponding to the column of the element; or, each element in the communication wait time matrix represents the communication wait time for the computing node corresponding to the column of the element to receive a word from the computing node corresponding to the row of the element. The step of performing outlier analysis on the communication wait time information set to identify abnormal communication wait times includes: Outlier analysis is performed on the communication wait time matrix to identify anomalous communication wait times.

[0082] In some embodiments, performing outlier analysis on the communication wait time matrix to identify anomalous communication wait times from the communication wait time matrix includes: Outlier analysis is performed on the rows, columns, and / or elements of the communication wait time matrix to identify anomalous communication wait times from the communication wait time matrix.

[0083] In some embodiments, outlier analysis is performed on rows in the communication wait time matrix to identify anomalous communication wait times from the communication wait time matrix, including: For each row in the communication waiting time matrix, calculate the average communication waiting time in that row, which is used as the average communication waiting time corresponding to that row. Outlier analysis is performed on the average communication wait time corresponding to each row of the communication wait time matrix to identify abnormal rows from the communication wait time matrix.

[0084] In some embodiments, locating the root cause of the performance anomaly of the hybrid expert model based on the abnormal communication wait time includes: If each element in the communication waiting time matrix represents the time taken for the computing node corresponding to the row of the element to wait for the computing node corresponding to the column of the element to receive a word, then the root cause of the performance anomaly of the hybrid expert model is located as a slow computing or congestion problem in the receiving node corresponding to the abnormal row. If each element in the communication waiting time matrix represents the time taken for the computing node corresponding to the column of the element to wait for the computing node corresponding to the row of the element to receive a word, then the root cause of the performance anomaly of the hybrid expert model is located as a problem of slow computation leading to transmission delay in the sending node corresponding to the abnormal row.

[0085] In some embodiments, the outlier analysis performed on the average communication wait time corresponding to each row of the communication wait time matrix to identify outlier rows from the communication wait time matrix includes: Calculate the average of the average communication waiting time corresponding to each row in the communication waiting time matrix, and calculate the standard deviation of the average communication waiting time corresponding to each row in the communication waiting time matrix; For each row in the communication waiting time matrix, calculate the ratio of the difference between the average communication waiting time corresponding to that row and the average value to the standard deviation. Rows whose ratio is greater than the threshold are identified as abnormal rows.

[0086] In some embodiments, outlier analysis is performed on columns of the communication wait time matrix to identify anomalous communication wait times from the communication wait time matrix, including: For each column in the communication waiting time matrix, calculate the average communication waiting time in that column, which is used as the average communication waiting time corresponding to that column. Outlier analysis is performed on the average communication wait time corresponding to each column in the communication wait time matrix to identify outlier columns from the communication wait time matrix.

[0087] In some embodiments, locating the root cause of the performance anomaly of the hybrid expert model based on the abnormal communication wait time includes: In the case that each element in the communication waiting time matrix represents the time taken for the computing node corresponding to the row of the element to wait for the computing node corresponding to the column of the element to receive a word, the root cause of the performance anomaly of the hybrid expert model is located as the sending node corresponding to the anomaly column having a slow computing problem that causes a sending delay. If each element in the communication waiting time matrix represents the time taken for the computing node corresponding to the column of the element to wait for the computing node corresponding to the row of the element to receive a term, then the root cause of the performance anomaly of the hybrid expert model is located as a slow computing or congestion problem in the receiving node corresponding to the anomaly column.

[0088] In some embodiments, the outlier analysis performed on the average communication wait time corresponding to each column in the communication wait time matrix to identify outlier columns from the communication wait time matrix includes: Calculate the average value of the average communication waiting time corresponding to each column in the communication waiting time matrix, and calculate the standard deviation of the average communication waiting time corresponding to each column in the communication waiting time matrix; For each column in the communication waiting time matrix, calculate the ratio of the difference between the average communication waiting time corresponding to the column and the average value to the standard deviation. Columns with ratios greater than the threshold are identified as abnormal columns.

[0089] In some embodiments, locating the root cause of the performance anomaly of the hybrid expert model based on the abnormal communication wait time includes: The link between the computing node corresponding to the row containing the abnormal communication wait time and the computing node corresponding to the column containing the abnormal communication wait time is found to be abnormal.

[0090] In some embodiments, outlier analysis is performed on the elements of the communication wait time matrix to identify anomalous communication wait times from the communication wait time matrix, including: Calculate the average value of the communication waiting time represented by each element in the communication waiting time matrix, and calculate the standard deviation of the communication waiting time represented by each element in the communication waiting time matrix; For each element in the communication waiting time matrix, calculate the ratio of the difference between the communication waiting time represented by the element and the average value to the standard deviation. The communication wait time represented by the element whose ratio is greater than the threshold is determined as an abnormal communication wait time.

[0091] In some embodiments, the time taken for the computing node to wait to receive word units from the target computing node is the average time taken for the computing node to wait to receive each word unit from the target computing node during one inference or training process of the hybrid expert model; or... The time taken for the computing node to wait for each word to be received from the target computing node is the average time taken for the computing node to wait for each word to be received from the target computing node during the data distribution phase of a single inference or training process of the hybrid expert model; or... The time taken for the computing node to wait to receive lexical units from the target computing node is the average time taken for the computing node to wait to receive each lexical unit from the target computing node during the result merging stage of a single inference or training process of the hybrid expert model.

[0092] For ease of description, the above devices are described by dividing them into various modules or units based on their functions. Of course, when implementing one or more of these specifications, the functions of each module or unit can be implemented in the same or different software and / or hardware, or a module that performs the same function can be implemented by a combination of multiple sub-modules or sub-units, etc. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0093] Based on the same concept as the methods described above, this specification also provides an electronic device, including: a processor; a memory for storing processor-executable instructions; wherein the processor performs the steps of the method as described in any of the above embodiments by executing the executable instructions.

[0094] Based on the same concept as the methods described above, this specification also provides a computer-readable storage medium having computer instructions stored thereon that, when executed by a processor, implement the steps of the methods as described in any of the above embodiments.

[0095] Based on the same concept as the methods described above, this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the methods as described in any of the above embodiments.

[0096] What those skilled in the art will understand is: In this specification, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, product, or apparatus. Without further limitation, the presence of additional identical or equivalent elements in a process, method, product, or apparatus that includes said elements is not excluded.

[0097] In this specification, “a,” “an,” and “the” do not specifically refer to the singular, but may also include the plural.

[0098] In this specification, ordinal numbers such as "first," "second," etc., do not necessarily indicate order; they are often used to distinguish between objects. For example, "first server" and "second server" usually refer to two servers. To differentiate between these two servers, they are described as "first server" and "second server." Of course, sometimes these two servers may be the same server.

[0099] In this specification, unless explicitly stated otherwise, "receiving and sending data" does not necessarily mean direct receiving and sending; it can also mean indirect receiving and sending. For example, A receiving data sent by B can be understood as A directly receiving the data sent by B, or it can be understood as A indirectly receiving the data sent by B through other entities such as C. Similarly, B sending data to A can be understood as B sending the data directly to A, or it can be understood as B indirectly sending the data to A through other entities such as C. Here, C can be one entity, or it can be two or more entities.

[0100] In this specification, unless explicitly stated otherwise, the relationships between structures can be direct or indirect. For example, when describing "A is connected to B," unless it is explicitly stated that A and B are directly connected, it should be understood that A can be directly connected to B or indirectly connected to B. Similarly, when describing "A is on top of B," unless it is explicitly stated that A is directly above B (AB is adjacent and A is above B), it should be understood that A can be directly above B or indirectly above B (AB is separated by other elements, and A is above B). And so on.

[0101] This specification uses specific terms to describe embodiments thereof. Terms such as "an embodiment," "one embodiment," and / or "some embodiments" refer to a particular feature, structure, or characteristic associated with at least one embodiment of this specification. Therefore, it should be emphasized and noted that references to "an embodiment," "one embodiment," or "an alternative embodiment" in different locations throughout this specification do not necessarily refer to the same embodiment. Furthermore, those skilled in the art can combine and integrate the different embodiments or examples described herein, as well as the features of those different embodiments or examples, without contradiction.

[0102] Although one or more embodiments of this specification provide method steps as described in the embodiments or flowcharts, it is understood that the order of steps listed in the embodiments or flowcharts is only one of many possible execution orders and does not represent the only execution order. Therefore, when the claims involve method steps, any changes or adjustments to the order of such steps, or the parallelism between steps, are also within the scope of protection of the claims.

Claims

1. A performance root cause analysis method using a hybrid expert model; wherein, The hybrid expert model is deployed in a distributed manner across several computing nodes; Any one of the computing nodes is determined as the master node. The method is applied to the master node and includes: Obtain communication waiting time information periodically sent by each computing node; wherein, the communication waiting time information includes the communication waiting time for the computing node to wait to receive tokens from each computing node; The communication wait time information sent by each computing node is aggregated into a communication wait time information set; Outlier analysis is performed on the communication wait time information set to identify abnormal communication wait times from the communication wait time information set; Based on the abnormal communication waiting time, the root cause of the performance anomaly of the hybrid expert model is located.

2. The method according to claim 1, wherein aggregating the communication waiting time information sent by each computing node into a communication waiting time information set includes: Based on the communication wait time information sent by each computing node, a communication wait time matrix is ​​constructed; wherein each element in the communication wait time matrix represents the communication wait time for the computing node corresponding to the row of the element to receive a word from the computing node corresponding to the column of the element; or, each element in the communication wait time matrix represents the communication wait time for the computing node corresponding to the column of the element to receive a word from the computing node corresponding to the row of the element. The step of performing outlier analysis on the communication wait time information set to identify abnormal communication wait times includes: Outlier analysis is performed on the communication wait time matrix to identify anomalous communication wait times.

3. The method according to claim 2, wherein performing outlier analysis on the communication waiting time matrix to determine abnormal communication waiting times from the communication waiting time matrix comprises: Outlier analysis is performed on the rows, columns, and / or elements of the communication wait time matrix to identify anomalous communication wait times from the communication wait time matrix.

4. The method according to claim 3, wherein outlier analysis is performed on rows in the communication wait time matrix to determine abnormal communication wait times from the communication wait time matrix, comprising: For each row in the communication waiting time matrix, calculate the average communication waiting time in that row, which is used as the average communication waiting time corresponding to that row. Outlier analysis is performed on the average communication wait time corresponding to each row of the communication wait time matrix to identify abnormal rows from the communication wait time matrix.

5. The method according to claim 4, wherein locating the root cause of the performance anomaly of the hybrid expert model based on the abnormal communication waiting time includes: If each element in the communication waiting time matrix represents the time taken for the computing node corresponding to the row of the element to wait for the computing node corresponding to the column of the element to receive a word, then the root cause of the performance anomaly of the hybrid expert model is located as a slow computing or congestion problem in the receiving node corresponding to the abnormal row. If each element in the communication waiting time matrix represents the time taken for the computing node corresponding to the column of the element to wait for the computing node corresponding to the row of the element to receive a word, then the root cause of the performance anomaly of the hybrid expert model is located as a problem of slow computation leading to transmission delay in the sending node corresponding to the abnormal row.

6. The method according to claim 4, wherein performing outlier analysis on the average communication wait time corresponding to each row of the communication wait time matrix to identify outlier rows from the communication wait time matrix includes: Calculate the average of the average communication waiting time corresponding to each row in the communication waiting time matrix, and calculate the standard deviation of the average communication waiting time corresponding to each row in the communication waiting time matrix; For each row in the communication waiting time matrix, calculate the ratio of the difference between the average communication waiting time corresponding to that row and the average value to the standard deviation. Rows whose ratio is greater than the threshold are identified as abnormal rows.

7. The method according to claim 3, wherein outlier analysis is performed on columns of the communication wait time matrix to determine abnormal communication wait times from the communication wait time matrix, comprising: For each column in the communication waiting time matrix, calculate the average communication waiting time in that column, which is used as the average communication waiting time corresponding to that column. Outlier analysis is performed on the average communication wait time corresponding to each column in the communication wait time matrix to identify outlier columns from the communication wait time matrix.

8. The method according to claim 7, wherein locating the root cause of the performance anomaly of the hybrid expert model based on the abnormal communication waiting time includes: In the case that each element in the communication waiting time matrix represents the time taken for the computing node corresponding to the row of the element to wait for the computing node corresponding to the column of the element to receive a word, the root cause of the performance anomaly of the hybrid expert model is located as the sending node corresponding to the anomaly column having a slow computing problem that causes a sending delay. If each element in the communication waiting time matrix represents the time taken for the computing node corresponding to the column of the element to wait for the computing node corresponding to the row of the element to receive a term, then the root cause of the performance anomaly of the hybrid expert model is located as a slow computing or congestion problem in the receiving node corresponding to the anomaly column.

9. The method according to claim 7, wherein performing outlier analysis on the average communication waiting time corresponding to each column in the communication waiting time matrix to identify outlier columns from the communication waiting time matrix comprises: Calculate the average value of the average communication waiting time corresponding to each column in the communication waiting time matrix, and calculate the standard deviation of the average communication waiting time corresponding to each column in the communication waiting time matrix; For each column in the communication waiting time matrix, calculate the ratio of the difference between the average communication waiting time corresponding to the column and the average value to the standard deviation. Columns with ratios greater than the threshold are identified as abnormal columns.

10. The method according to claim 4, wherein locating the root cause of the performance anomaly of the hybrid expert model based on the abnormal communication waiting time includes: The link between the computing node corresponding to the row containing the abnormal communication wait time and the computing node corresponding to the column containing the abnormal communication wait time is found to be abnormal.

11. The method according to claim 4, wherein outlier analysis is performed on the elements in the communication wait time matrix to determine abnormal communication wait times from the communication wait time matrix, comprising: Calculate the average value of the communication waiting time represented by each element in the communication waiting time matrix, and calculate the standard deviation of the communication waiting time represented by each element in the communication waiting time matrix; For each element in the communication waiting time matrix, calculate the ratio of the difference between the communication waiting time represented by the element and the average value to the standard deviation. The communication wait time represented by the element whose ratio is greater than the threshold is determined as an abnormal communication wait time.

12. The method according to claim 1, wherein the time consumed by the computing node to wait for receiving word units from the target computing node is the average time consumed by the computing node to wait for receiving each word unit from the target computing node during one inference or training process of the hybrid expert model; or, The time taken for the computing node to wait to receive word units from the target computing node is the average time taken for the computing node to wait to receive each word unit from the target computing node during the data distribution phase of a single inference or training process of the hybrid expert model. or, The time taken for the computing node to wait to receive lexical units from the target computing node is the average time taken for the computing node to wait to receive each lexical unit from the target computing node during the result merging stage of a single inference or training process of the hybrid expert model.

13. A performance root cause analysis device using a hybrid expert model; wherein, The hybrid expert model is deployed in a distributed manner across several computing nodes; Any one of the computing nodes is determined as the master node. The device is applied to the master node and includes: The acquisition module acquires communication waiting time information periodically sent by each computing node; wherein, the communication waiting time information includes the communication waiting time for the computing node to wait to receive tokens from each computing node; The aggregation module aggregates the communication wait time information sent by each computing node into a communication wait time information set; The analysis module performs outlier analysis on the communication waiting time information set to identify abnormal communication waiting times from the communication waiting time information set; The localization module locates the root cause of the performance anomaly of the hybrid expert model based on the abnormal communication waiting time.

14. An electronic device comprising: processor; Memory used to store processor-executable instructions; The processor implements the steps of the method as described in any one of claims 1 to 12 by executing the executable instructions.

15. A computer-readable storage medium having stored thereon computer instructions that, when executed by a processor, implement the steps of the method as claimed in any one of claims 1 to 12.

16. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method as claimed in any one of claims 1 to 12.

Images