Elastic offloading expert parallel acceleration method and device based on global base matrix

Abstract

The application provides an elastic shunting expert parallel acceleration method and device based on a global base matrix, which comprises the following steps: obtaining input information and determining the target expert and the computing device where the target expert is located; monitoring the real-time congestion state of the device and judging whether the congestion condition is triggered; if the congestion condition is triggered, intercepting the information transmission, starting a virtual expert generation process in the current device, extracting the corresponding coefficient vector from a local shared base matrix set, and combining the base matrix to dynamically reconstruct a virtual expert weight matrix; using the weight matrix to perform local forward calculation on the information to generate a first result and pass the first result to the next layer, while reducing the dimension of the information to generate a reduced dimension feature vector; and placing the reduced dimension feature vector into a low-priority background communication queue and sending the reduced dimension feature vector to the target device asynchronously for residual compensation calculation, so as to effectively avoid network bottlenecks and significantly improve the overall throughput and computing efficiency of the system.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an elastic shunt expert parallel acceleration method and apparatus based on a global basis matrix. Background Technology

[0002] In the field of artificial intelligence, Mixture of Experts (MoE) models, with their sparse activation characteristics, have become one of the mainstream architectures for building ultra-large-scale parametric models. To support efficient training and inference of MoE models on massive datasets, the industry widely adopts the Expert Parallelism (EP) strategy. This strategy distributes different expert networks across multiple computing devices (such as GPUs), utilizing all-to-all communication to route and distribute information among different experts. As model and cluster sizes continue to expand, effectively reducing cross-node communication overhead, improving device utilization, and maintaining the stability and throughput of large-scale distributed training have become core needs that urgently require solutions in this field.

[0003] To address these needs, the relevant technologies are optimized primarily in two directions. First, dynamic load balancing optimization: by monitoring the call rate and computation time of each expert in real time, setting load thresholds and dynamically distributing tokens from overloaded experts to underloaded experts, or by dynamically increasing or decreasing the number of experts, introducing gradient balancing and expert regularization, the resource unevenness caused by hot and cold experts is alleviated. Second, routing algorithm upgrades: by having experts actively select tokens, adaptively adjusting the number of experts allocated (dynamic K value), and introducing regularization and smoothing functions to suppress routing noise, the matching accuracy between tokens and experts is improved, reducing invalid routes.

[0004] However, the aforementioned technologies still suffer from the following core problems: Token routing still relies on All-to-All communication, resulting in high cross-node transmission latency; load balancing schemes require real-time collection of status information, increasing computational and scheduling overhead, and suffer from data lag, making it difficult to completely resolve the uneven device utilization caused by hot and cold experts; upgraded routing algorithms further increase the computational pressure on gating networks and route matching, and have poor generalization capabilities. More fundamentally, in the parallel architecture of related experts, tokens must wait for the physical GPU of their target expert to complete communication before computation can begin. When congestion occurs on the target GPU or network path, the computation process is constrained by communication blocking, limiting the overall load balancing capability and latency performance of the system. Therefore, there is an urgent need for a method that can utilize idle computing resources in the cluster to process tokens sent to overloaded experts in real time without waiting for cross-node communication to complete, thereby breaking the blockage of the computation process by communication congestion. Summary of the Invention

[0005] This application provides an expert parallel acceleration method and apparatus for elastic load balancing based on a global basis matrix, in order to solve the problem of low system efficiency caused by physical network congestion and waiting for computing resources.

[0006] The first aspect of this application provides a parallel acceleration method for elastic flow splitting experts based on a global basis matrix, the method comprising: Acquire the input information to be processed, and determine the target expert of the input information and the target computing device where the target expert is located; Monitor the real-time congestion status of the target computing device, and determine whether a congestion condition is triggered based on the real-time congestion status; When the congestion condition is determined to be triggered, the transmission of the input information is intercepted, and a virtual expert generation process is initiated in the current computing device; Extract the coefficient vector corresponding to the target expert from the locally stored set of shared basis matrices, and dynamically reconstruct the virtual expert weight matrix using the shared basis matrices and the coefficient vector; The input information is processed using the virtual expert weight matrix to generate a first calculation result; The first calculation result is passed to the next layer calculation operator, and the input information is subjected to dimensionality reduction processing to generate a dimensionality-reduced feature vector; The dimensionality-reduced feature vector is sent to the target computing device via background communication so that the target computing device can perform residual compensation calculation.

[0007] This solution monitors the congestion status of the target computing device and triggers congestion conditions, intercepts the transmission of input information, initiates a virtual expert generation process in the current computing device, dynamically reconstructs the virtual expert weight matrix using the locally stored shared basis matrix and coefficient vector, performs local forward calculation, passes the first calculation result to the next layer computing operator, and simultaneously generates a dimensionality-reduced feature vector and asynchronously sends it to the target computing device for residual compensation calculation. This alleviates the blockage of the computing process by physical network congestion, reduces cross-node communication waiting latency, and improves the overall load balance and computing efficiency of the system.

[0008] Optionally, before acquiring the input information to be processed, the method further includes: Obtain the multiple original expert weight matrices in the hybrid expert model; The multiple original expert weight matrices are mathematically decomposed to extract the shared basis matrix and the unique coefficient vector and residual matrix corresponding to each original expert weight matrix; The shared basis matrix is ​​distributed and stored in each computing device in the distributed system, and the unique coefficient vector corresponding to each original expert weight matrix is ​​associated with the corresponding expert identification information and stored.

[0009] Before acquiring the input information to be processed, this scheme performs mathematical decomposition on multiple original expert weight matrices in the hybrid expert model to extract the shared basis matrix and the unique coefficient vector and residual matrix corresponding to each original expert weight matrix. Then, the shared basis matrix is ​​distributed and stored to each computing device in the distributed system, and the unique coefficient vector corresponding to each original expert weight matrix is ​​associated with the corresponding expert identification information for storage. This provides a pre-set shared basis and coefficient vector for subsequent dynamic reconstruction of virtual experts on the local computing devices, reduces the communication overhead of full cross-node transmission of expert weights, and alleviates the storage redundancy problem caused by expert weight copying.

[0010] Optionally, monitoring the real-time congestion status of the target computing device and determining whether a congestion condition is triggered based on the real-time congestion status includes: Read the hardware congestion register of the underlying network interconnection protocol to obtain the network interface card send queue depth or network link utilization of the target computing device; Compare the sending queue depth with a preset congestion threshold, or determine whether the network link utilization exceeds a preset bandwidth limit; When the sending queue depth is greater than the preset congestion threshold, or the network link utilization exceeds the preset bandwidth limit, the congestion condition is determined to be triggered.

[0011] This solution obtains the transmission queue depth or network link utilization of the target computing device's network interface card by reading the hardware congestion register of the underlying network interconnection protocol. It then compares the transmission queue depth with a preset congestion threshold or determines whether the network link utilization exceeds a preset bandwidth limit. When the transmission queue depth exceeds the preset congestion threshold or the network link utilization exceeds the preset bandwidth limit, a congestion condition is triggered. This enables real-time, low-level hardware-level awareness of the network congestion status of the target computing device, providing accurate triggering basis for subsequent transmission interception and triggering local virtual expert computation. This reduces the lag of software prediction or load sampling-based methods and improves the timeliness and accuracy of congestion detection.

[0012] Optionally, the step of dynamically reconstructing the virtual expert weight matrix using the shared basis matrix and the coefficient vector includes: Call the tensor computation core unit in the current computing device; Load the shared basis matrix and obtain the specific coefficient vector associated with the target expert; A linear weighting operation is performed to perform a dot product or weighted summation between the specific coefficient vector and the shared basis matrix to generate the virtual expert weight matrix, which is a low-rank approximation of the target expert weight matrix.

[0013] This solution calls the tensor computation core unit in the current computing device, loads the shared basis matrix, obtains the specific coefficient vector associated with the target expert, and then performs a linear weighting operation to perform a dot product or weighted summation on the specific coefficient vector and the shared basis matrix to generate a virtual expert weight matrix. This virtual expert weight matrix is ​​a low-rank approximation of the target expert weight matrix, thereby quickly synthesizing virtual expert weights for processing input information locally on the current computing device. This reduces the dependence on the transmission of full expert weights across nodes, reduces physical network bandwidth consumption, and improves the efficiency of local computation.

[0014] Optionally, the method further includes: When it is determined that the congestion condition has not been triggered, the input information is pushed into the regular communication transmission queue; The input information is transmitted to the target computing device via a high-speed interconnection network so that real experts in the target computing device can perform calculations.

[0015] When the congestion condition is not triggered, this solution pushes the input information into the regular communication transmission queue and transmits it to the target computing device through a high-speed interconnection network. This allows real experts on the target computing device to perform calculations, thus maintaining the original parallel communication and computing path of experts when the network is in normal condition. This ensures that the standard execution process of the sparse model is not disturbed, and enables dynamic switching between local computation in congestion scenarios and physical transmission in normal scenarios. This improves the system's adaptive adjustment capability under different network load conditions.

[0016] Optionally, the step of performing dimensionality reduction processing on the input information to generate a dimensionality-reduced feature vector includes: Construct a dimension-reduced projection transformation matrix, the dimension of which is much smaller than the feature dimension of the original input information; The feature vector of the input information is multiplied by the dimension reduction projection transformation matrix to generate a compressed feature vector with extremely low dimension, the dimension of which is one-sixteenth or less of the original feature dimension.

[0017] This scheme constructs a dimension-reduced projection transformation matrix with a dimension much smaller than the feature dimension of the original input information. Then, it performs matrix multiplication on the feature vector of the input information and the dimension-reduced projection transformation matrix to generate a compressed feature vector with an extremely low dimension, which is one-sixteenth or less of the original feature dimension. This transforms the input information processed by the local virtual expert into a lightweight compressed representation, reduces the amount of data in asynchronous background communication, reduces network bandwidth usage, alleviates congestion pressure on high-priority communication traffic, and improves data transmission efficiency in the residual compensation stage.

[0018] Optionally, sending the dimensionality-reduced feature vector to the target computing device via background communication for the target computing device to perform residual compensation calculation includes: Detect the idle state or fragmented transmission window of the network link; When available bandwidth is detected on the network link, an asynchronous data packet carrying the reduced-dimensional feature vector is sent. During the intervals between completing the main computing tasks, the target computing device uses the reduced-dimensional feature vector and the pre-stored residual matrix to calculate the residual compensation amount, and then fuses the residual compensation amount with the first calculation result to restore the computing accuracy.

[0019] This solution detects the idle state or fragmented transmission window of the network link. When available bandwidth is detected, it sends asynchronous data packets carrying dimensionality-reduced feature vectors. During the intervals between completing the main computing tasks, the target computing device uses the dimensionality-reduced feature vectors and pre-stored residual matrices to calculate the residual compensation amount. The residual compensation amount is then fused with the first calculation result to restore the calculation accuracy. This fully utilizes the idle bandwidth of the network and the idle computing intervals of the computing device to complete the accuracy closed loop, reduces the interference of the asynchronous compensation process on the main communication path and the main computing task, and improves the overall resource utilization efficiency of the system and the accuracy of the final calculation results.

[0020] Optionally, the method is applicable to expert parallel computing scenarios between computing units within a graphics processor, between different graphics processors, or between different computing nodes.

[0021] This solution applies the method to expert parallel computing scenarios between computing units within a graphics processor, between different graphics processors, or between different computing nodes. This allows for the unified application of congestion awareness, local virtual expert reconstruction, and asynchronous compensation mechanisms at distributed computing levels of different granularities. It expands the applicability of the method in various hardware topologies such as multiple computing units within a single card, multi-card interconnection, and cross-node clusters, and improves the compatibility and portability of expert parallel acceleration solutions under different deployment scales.

[0022] Optionally, the shared basis matrix is ​​extracted from multiple original expert weight matrices by singular value decomposition or principal component analysis. The shared basis matrix has orthogonal properties and is configured to reside in the static random access memory or secondary cache of each computing device.

[0023] In this scheme, the shared basis matrix is ​​extracted from multiple original expert weight matrices through singular value decomposition or principal component analysis. The shared basis matrix has orthogonal properties and is configured to reside in the static random access memory or L2 cache of each computing device. This allows for the acquisition of a high-precision shared representation in a mathematically optimal low-rank decomposition manner. The orthogonal basis matrix is ​​stored at the cache level, reducing the data read latency during the local virtual expert reconstruction process and improving the dynamic generation speed of the virtual expert weight matrix.

[0024] A second aspect of this application provides a resilient shunt expert parallel acceleration device based on a global basis matrix, applied to the resilient shunt expert parallel acceleration method based on a global basis matrix described in the first aspect. The device includes: An information acquisition module is used to acquire input information to be processed and to determine the target expert of the input information and the target computing device where the target expert is located; The congestion monitoring module is used to monitor the real-time congestion status of the target computing device and determine whether a congestion condition is triggered based on the real-time congestion status. The traffic control module is used to intercept the transmission of the input information when the congestion condition is determined to be triggered, and to start the virtual expert generation process in the current computing device. The virtual expert construction module is used to extract the coefficient vector corresponding to the target expert from the locally stored set of shared basis matrices, and dynamically reconstruct the virtual expert weight matrix using the shared basis matrices and the coefficient vector; A local computing module is used to perform local forward computation on the input information using the virtual expert weight matrix to generate a first computation result; An asynchronous communication module is used to pass the first calculation result to the next layer calculation operator, perform dimensionality reduction processing on the input information to generate a dimensionality-reduced feature vector, and send the dimensionality-reduced feature vector to the target computing device through a background communication method so that the target computing device can perform residual compensation calculation.

[0025] The second aspect of this application provides a flexible expert parallel acceleration device based on a global basis matrix. This device uses an information acquisition module to determine the target expert and target computing device for the input information, a congestion monitoring module to monitor real-time congestion status and determine if congestion conditions are triggered, a traffic control module to intercept transmission and initiate a virtual expert generation process when congestion is triggered, a virtual expert construction module to extract coefficient vectors from local storage and dynamically reconstruct the virtual expert weight matrix, a local computing module to perform local forward computation using this matrix to generate a first computation result, and an asynchronous communication module to pass the first computation result to the next layer operator and, after dimensionality reduction of the input information, to place it in a low-priority background queue and asynchronously send it to the target computing device for residual compensation computation. This achieves complete device-level process support from physical transmission to local computation in congestion scenarios, reducing cross-node communication latency and improving the efficiency of expert parallel computation.

[0026] As can be seen from the above technical solutions, this application provides a method and apparatus for elastic flow splitting expert parallel acceleration based on a global basis matrix. This method involves acquiring input information to be processed and determining the target expert and the target computing device where the target expert resides; monitoring the real-time congestion status of the target computing device and determining whether a congestion condition is triggered based on the real-time congestion status; intercepting the transmission of the input information when the congestion condition is triggered and initiating a virtual expert generation process in the current computing device; extracting the coefficient vector corresponding to the target expert from a locally stored shared basis matrix set and dynamically reconstructing the virtual expert weight matrix using the shared basis matrix and the coefficient vector; performing local forward computation on the input information using the virtual expert weight matrix to generate a first computation result; passing the first computation result to the next-layer computation operator and performing dimensionality reduction processing on the input information to generate a dimensionality-reduced feature vector; and sending the dimensionality-reduced feature vector to the target computing device via background communication for the target computing device to perform residual compensation computation, thereby solving the problem of low system efficiency caused by physical network congestion and waiting for computing resources. Attached Figure Description

[0027] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a flowchart illustrating the expert parallel acceleration method for elastic flow splitting based on a global basis matrix, provided in an embodiment of this application. Detailed Implementation

[0029] The embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described below do not represent all embodiments consistent with this application. They are merely examples of systems and methods consistent with some aspects of this application as detailed in the claims.

[0030] To address the system inefficiency caused by physical network congestion and waiting for computing resources, see [link to relevant documentation]. Figure 1 This application provides a parallel acceleration method for elastic load balancing experts based on a global basis matrix, the method including: S100: Acquire the input information to be processed, and determine the target expert of the input information and the target computing device where the target expert is located.

[0031] It should be understood that "input information (such as a token)" refers to the feature vector after processing by the previous layer of the network, and the "target expert" is determined based on the routing decision calculated by the gating network.

[0032] In some embodiments, before acquiring the input information to be processed, the method further includes: Obtain the multiple original expert weight matrices in the hybrid expert model; Mathematical decomposition is performed on multiple original expert weight matrices to extract the shared basis matrix and the unique coefficient vector and residual matrix corresponding to each original expert weight matrix; The shared basis matrix is ​​distributed and stored on each computing device in the distributed system, and the unique coefficient vector corresponding to each original expert weight matrix is ​​associated with the corresponding expert identification information and stored.

[0033] S200: Monitors the real-time congestion status of the target computing device and determines whether a congestion condition is triggered based on the real-time congestion status.

[0034] In some embodiments, monitoring the real-time congestion status of the target computing device and determining whether a congestion condition is triggered based on the real-time congestion status includes: Read the hardware congestion register of the underlying network interconnection protocol to obtain the network interface card send queue depth or network link utilization of the target computing device; Compare the sending queue depth with a preset congestion threshold, or determine whether the network link utilization exceeds the preset bandwidth limit; When the sending queue depth is greater than the preset congestion threshold, or the network link utilization exceeds the preset bandwidth limit, a congestion condition is determined to be triggered.

[0035] It should be understood that the underlying network interconnection protocol usually refers to high-speed network protocols such as InfiniBand and RoCE, which are suitable for high-performance distributed computing scenarios. The hardware network cards of these protocols natively support the output of congestion status registers, which can be read directly without additional software sampling and statistics, and can reduce the monitoring latency to the nanosecond level. The sending queue depth refers to the number of data packets currently queued for data to be sent by the network interface card of the target computing device. In this embodiment, the preset congestion threshold can be configured according to the actual hardware communication bandwidth, and the typical value range is set in [32, 128] data packets. When the number of data packets to be sent by a single card exceeds the set value corresponding to this range, it can be determined that the current hardware communication has entered a congested state. For the scenario of judging network link utilization, the preset bandwidth limit is usually set to 80% to 90% of the total physical link bandwidth, reserving 10% to 20% of bursty redundant bandwidth to avoid occasional data conflicts being mistakenly judged as congestion.

[0036] S300: When a congestion condition is determined to be triggered, the transmission of input information is intercepted, and a virtual expert generation process is initiated in the current computing device.

[0037] It should be understood that intercepting the transmission of input information does not mean directly discarding the computational task corresponding to the data packet. Instead, it temporarily intercepts the task, which would normally need to be sent to the corresponding expert unit on another device, and stores it in the local task cache queue of the current device. This prevents the already congested network link from further increasing the transmission load and exacerbating the congestion problem. The virtual expert generation process utilizes the idle computing power of the current computing device to generate a temporary virtual expert unit locally. This unit processes the intercepted task directly on the local machine, without sending the task over the network to a remote physical expert unit. This bypasses the congested network link while ensuring task processing efficiency and preventing task processing delays.

[0038] S400: Extracts the coefficient vector corresponding to the target expert from the locally stored set of shared basis matrices, and dynamically reconstructs the virtual expert weight matrix using the shared basis matrices and coefficient vectors.

[0039] In some embodiments, the virtual expert weight matrix is ​​dynamically reconstructed using a shared basis matrix and coefficient vector, including: Call the tensor computation core unit in the current computing device; Load the shared basis matrix and obtain the specific coefficient vector associated with the target expert; Perform linear weighting operations to generate a virtual expert weight matrix by dot product or weighted summation of a specific coefficient vector with a shared basis matrix. The virtual expert weight matrix is ​​a low-rank approximation of the target expert weight matrix.

[0040] It should be understood that low-rank approximation refers to the fact that the rank of the reconstructed weight matrix is ​​much smaller than the rank of the original complete weight matrix. Since the shared basis matrix is ​​shared by all target experts, it only needs to be stored once, eliminating the need for each expert to store the complete weight matrix, thus significantly reducing storage resource consumption. In this embodiment, the number of bases of the orthogonal basis matrix can be preset according to the target hardware memory capacity, generally set to 1 / 5 to 1 / 10 of the rank of the original expert weight matrix. For example, in a scenario where the rank of the original complete weight matrix is ​​1000, the number of bases of the shared orthogonal basis matrix can be set to 100 to 200, maximizing storage compression while ensuring acceptable loss of inference accuracy. The tensor computation core unit here is a dedicated processing unit that supports high-speed parallel tensor computation in current computing devices. It can be directly called through the underlying programming interface provided by the device, and the latency of completing the matrix dot product operation is generally less than 1 microsecond, which will not cause additional perceptible time consumption to the overall inference process.

[0041] S500: Utilizes a virtual expert weight matrix to perform local forward computation on the input information to generate the first computation result.

[0042] It should be understood that the virtual expert weight matrix is ​​obtained by low-rank approximation decomposition of the original expert weight matrix, and reconstructed by multiplying the globally shared orthogonal basis matrix and the corresponding expert's private low-rank projection matrix. Therefore, it is not necessary to store the complete original weight matrix for each expert separately; only a copy of the global orthogonal basis matrix and a small-sized private projection matrix corresponding to each expert need to be stored, thus significantly reducing memory and GPU memory usage. The input information here refers to the Transformer model input feature tensor after pre-embedding encoding. Local forward computation refers to the computation process performed entirely within the current computing device, through the tensor computation core unit, without requiring cross-device data transmission and interaction. The final generated first computation result is the input feature output result extracted by the corresponding layer of the virtual expert, which can be directly supplied to subsequent network layers for further processing.

[0043] S600: Pass the first calculation result to the next layer calculation operator and perform dimensionality reduction processing on the input information to generate a dimensionality-reduced feature vector.

[0044] In some embodiments, the input information is subjected to dimensionality reduction processing to generate a dimensionality-reduced feature vector, including: Construct a dimension-reduced projection transformation matrix, the dimension of which is much smaller than the feature dimension of the original input information; The feature vector of the input information is multiplied by the dimension reduction projection transformation matrix to generate a compressed feature vector with extremely low dimension. The dimension of the compressed feature vector is one-sixteenth or less of the dimension of the original feature vector.

[0045] The dimension reduction projection transformation matrix is ​​a global matrix shared by experts across all MoE layers. It can be pre-stored in the GPU memory of the computing device, eliminating the need for repeated allocation of storage resources with each computation batch. The number of rows in the dimension reduction projection transformation matrix is ​​determined by the number of bases in the orthogonal basis matrix, ensuring that the compressed feature dimension can be directly used in subsequent matrix operations with the orthogonal basis matrix. The specific value of the dimension reduction dimension can also be preset according to the GPU memory capacity and accuracy requirements of the target hardware. In scenarios with limited GPU memory resources and lower requirements for inference accuracy, the compressed feature dimension can be set to 1 / 32 or 1 / 64 of the original feature dimension, further reducing the amount of data communicated across devices.

[0046] S700: Sends the dimensionality-reduced feature vector to the target computing device via background communication so that the target computing device can perform residual compensation calculation.

[0047] In some embodiments, the dimensionality-reduced feature vector is sent to the target computing device via a background communication method so that the target computing device can perform residual compensation calculation, including: Detect the idle state or fragmented transmission window of the network link; When available bandwidth is detected on the network link, an asynchronous data packet carrying a reduced-dimensional feature vector is sent. During the intervals between completing the main computing tasks, the target computing device uses the dimensionality-reduced feature vectors and pre-stored residual matrices to calculate the residual compensation amount, and then fuses the residual compensation amount with the first calculation result to restore the computing accuracy.

[0048] It should be understood that a network link idle state refers to a state where the current computing device has no high-priority real-time transmission tasks and there is remaining usable bandwidth in the link; a fragmented transmission window refers to an idle time window in the link that cannot satisfy the transmission of complete high-priority data and can only carry small batches of data transmission. This method puts the dimensionality-reduced feature vectors into a low-priority queue, which is specifically designed to utilize these scattered idle bandwidth resources to complete the transmission. It will not preempt the communication resources of high-priority core computing tasks, thus maximizing the utilization of network bandwidth and avoiding communication congestion that slows down the overall computing process.

[0049] The residual matrix in this method is also a global matrix shared by all MoE layer experts. It is pre-stored in the video memory of each target computing device and does not need to be repeatedly allocated storage resources with each computing batch. Its dimensions correspond to and match the dimensions of the compressed feature vector and the original feature vector. The formula for calculating the residual compensation amount is: residual compensation amount = residual matrix × compressed feature vector. The final fused complete calculation result is equal to the first calculation result plus the obtained residual compensation amount. In this way, the original calculation accuracy is restored through low-cost post-processing, balancing the needs of communication cost and calculation accuracy.

[0050] In some embodiments, the method further includes: When it is determined that no congestion condition has been triggered, the input information is pushed into the regular communication sending queue; Input information is transmitted to the target computing device via a high-speed interconnect network, so that real experts on the target computing device can perform calculations.

[0051] It should be understood that congestion specifically refers to the remaining available bandwidth of the current high-speed interconnection network being lower than a preset threshold. This preset threshold can be configured according to the communication priority of the overall cluster tasks, and is set by default to [X] times the minimum bandwidth required for a single input message. The regular communication sending queue is a queue used to carry high-priority, latency-sensitive expert computing task transmission requests. The transmission priority of this queue is higher than that of the low-priority queue used to transmit compressed feature vectors in this method. The system will prioritize satisfying the transmission needs of the regular communication sending queue during scheduling. The input information is a data unit carrying expert computing task allocation information and the original feature data to be calculated. After receiving the input information, the target computing device will first extract the corresponding task allocation identifier, and then schedule the original feature data in the input information to the corresponding real expert computing unit for calculation to obtain the final first calculation result.

[0052] In some embodiments, the method is applicable to expert parallel computing scenarios between computing units within a graphics processor, between different graphics processors, or between different computing nodes.

[0053] It should be understood that the solution of the present invention can be applied to interconnection scenarios of different computing cores within the same chip package, and can also be adapted to distributed computing cluster scenarios where multiple chips and multiple nodes are interconnected through high-speed buses and network switches. For different interconnection scenarios, those skilled in the art can adjust parameters such as preset bandwidth thresholds and compression basis dimensions according to the latency characteristics of actual hardware communication, and these adjustments do not deviate from the core method framework of the present invention.

[0054] In some embodiments, the shared basis matrix is ​​extracted from multiple original expert weight matrices by singular value decomposition or principal component analysis. The shared basis matrix has orthogonal properties and is configured to reside in the static random access memory or secondary cache of each computing device.

[0055] It should be understood that the extracted shared basis matrix can cover most of the key feature information contained in the original expert weight matrix. After the original feature vector is compressed by the basis projection transformation, it can retain a computational accuracy that is close enough to the original result while significantly reducing the dimension. Therefore, it will not have a significant impact on the accuracy of the final model inference. The orthogonal basis adopts a globally shared design, and all expert computation units reuse the same basis. There is no need to store different compression matrices for each expert. Therefore, the overall additional storage overhead of this method is very low. At the same time, the characteristics of the orthogonal basis can also ensure the numerical stability of the projection compression and decompression calculation process and avoid the accumulation of computational errors.

[0056] This application also provides a parallel acceleration device for elastic shunt expert based on a global basis matrix, applied to the parallel acceleration method for elastic shunt expert based on a global basis matrix provided in the above embodiments. The device includes: The information acquisition module is used to acquire the input information to be processed and to determine the target expert of the input information and the target computing device where the target expert is located.

[0057] It should be understood that when the information acquisition module determines the target expert for the input information, it first calculates the allocation probability of each registered expert corresponding to the input information based on the expert routing strategy of the large language model, selects the expert with the highest probability as the target expert, and then reads the device number corresponding to the target expert according to the preset mapping table between experts and computing devices to complete the location of the target computing device.

[0058] The congestion monitoring module is used to monitor the real-time congestion status of the target computing device and determine whether a congestion condition is triggered based on the real-time congestion status.

[0059] It should be understood that the real-time congestion status is the number of bits occupied by the hardware queue of the information requests currently pending on the target computing device. The congestion condition is preset to be greater than the preset queue threshold, which can be adjusted according to the graphics memory bandwidth and the average processing time per message of the computing device.

[0060] The traffic control module is used to intercept the transmission of input information when a congestion condition is determined to be triggered, and to start the virtual expert generation process in the current computing device.

[0061] It should be understood that intercepting the transmission of input information specifically involves: disabling the transmission enable bit of the corresponding target device's DMA transmission channel through the network card's traffic control unit, preventing the original data of the input information from being copied in batches to the target computing device's video memory space, and avoiding invalid data transmission from occupying bandwidth between devices.

[0062] The virtual expert construction module is used to extract the coefficient vector corresponding to the target expert from the locally stored set of shared basis matrices, and dynamically reconstruct the virtual expert weight matrix using the shared basis matrices and coefficient vectors.

[0063] It should be understood that the set of shared basis matrices is stored in the local cache of each computing device and is generated after the low-rank decomposition is completed during the training phase of the large language model. Each basis matrix satisfies the row orthogonality constraint, and the coefficient vector is the corresponding linear combination coefficient obtained from the low-rank decomposition of the corresponding expert. The virtual expert weight matrix after reconstruction satisfies W=V×α, where V is the shared basis matrix and α is the extracted coefficient vector. The expressive ability error between the reconstructed virtual expert and the original target expert meets the preset accuracy requirements.

[0064] The local computation module is used to perform local forward computation on the input information using the virtual expert weight matrix to generate the first computation result.

[0065] It should be understood that the first calculation result is the preliminary feature output of the virtual expert after processing the input information. Passing it to the next layer of calculation operators can skip the idle period of waiting for cross-device data, mask cross-device communication delays, and improve the utilization of computing devices.

[0066] The asynchronous communication module is used to pass the first calculation result to the next layer calculation operator, perform dimensionality reduction processing on the input information, generate dimensionality-reduced feature vectors, and send the dimensionality-reduced feature vectors to the target computing device through background communication so that the target computing device can perform residual compensation calculations.

[0067] It should be understood that the low-rank projection matrix is ​​a dimension reduction matrix generated in conjunction with the shared basis matrix. The dimension of the feature vector after dimension reduction is only [1 / K] of the dimension of the original input information, where K is a preset dimension reduction factor. This significantly reduces the amount of data transmitted asynchronously in the background. After the target computing device receives the dimension reduction feature vector, it will restore the residual input based on the low-rank projection matrix, calculate the residual compensation result, and then asynchronously transmit the compensation result back to the corresponding computing device to update the final output features, ensuring that the overall computational accuracy of the model is not lost.

[0068] Similar parts between the embodiments provided in this application can be referred to mutually. The specific implementation methods provided above are only a few examples under the overall concept of this application and do not constitute a limitation on the scope of protection of this application. For those skilled in the art, any other implementation methods extended from the solution of this application without creative effort shall fall within the scope of protection of this application.

Claims

1. A parallel acceleration method for elastic flow splitting experts based on a global basis matrix, characterized in that, The method includes: Acquire the input information to be processed, and determine the target expert of the input information and the target computing device where the target expert is located; Monitor the real-time congestion status of the target computing device, and determine whether a congestion condition is triggered based on the real-time congestion status; When the congestion condition is determined to be triggered, the transmission of the input information is intercepted, and a virtual expert generation process is initiated in the current computing device; Extract the coefficient vector corresponding to the target expert from the locally stored set of shared basis matrices, and dynamically reconstruct the virtual expert weight matrix using the shared basis matrices and the coefficient vector; The input information is processed using the virtual expert weight matrix to generate a first calculation result; The first calculation result is passed to the next layer calculation operator, and the input information is subjected to dimensionality reduction processing to generate a dimensionality-reduced feature vector; The dimensionality-reduced feature vector is sent to the target computing device via background communication so that the target computing device can perform residual compensation calculation.

2. The elastic flow splitting expert parallel acceleration method based on a global basis matrix according to claim 1, characterized in that, Before acquiring the input information to be processed, the method further includes: Obtain the multiple original expert weight matrices in the hybrid expert model; The multiple original expert weight matrices are mathematically decomposed to extract the shared basis matrix and the unique coefficient vector and residual matrix corresponding to each original expert weight matrix; The shared basis matrix is ​​distributed and stored in each computing device in the distributed system, and the unique coefficient vector corresponding to each original expert weight matrix is ​​associated with the corresponding expert identification information and stored.

3. The elastic flow splitting expert parallel acceleration method based on a global basis matrix according to claim 1, characterized in that, The monitoring of the real-time congestion status of the target computing device, and the determination of whether a congestion condition is triggered based on the real-time congestion status, includes: Read the hardware congestion register of the underlying network interconnection protocol to obtain the network interface card send queue depth or network link utilization of the target computing device; Compare the sending queue depth with a preset congestion threshold, or determine whether the network link utilization exceeds a preset bandwidth limit; When the sending queue depth is greater than the preset congestion threshold, or the network link utilization exceeds the preset bandwidth limit, the congestion condition is determined to be triggered.

4. The elastic flow splitting expert parallel acceleration method based on a global basis matrix according to claim 1, characterized in that, The dynamic reconstruction of the virtual expert weight matrix using the shared basis matrix and the coefficient vector includes: Call the tensor computation core unit in the current computing device; Load the shared basis matrix and obtain the specific coefficient vector associated with the target expert; A linear weighting operation is performed to perform a dot product or weighted summation between the specific coefficient vector and the shared basis matrix to generate the virtual expert weight matrix, which is a low-rank approximation of the target expert weight matrix.

5. The elastic flow splitting expert parallel acceleration method based on a global basis matrix according to claim 1, characterized in that, The method further includes: When it is determined that the congestion condition has not been triggered, the input information is pushed into the regular communication transmission queue; The input information is transmitted to the target computing device via a high-speed interconnection network so that real experts in the target computing device can perform calculations.

6. The elastic flow splitting expert parallel acceleration method based on a global basis matrix according to claim 1, characterized in that, The step of performing dimensionality reduction processing on the input information to generate a dimensionality-reduced feature vector includes: Construct a dimension-reduced projection transformation matrix, the dimension of which is much smaller than the feature dimension of the original input information; The feature vector of the input information is multiplied by the dimension reduction projection transformation matrix to generate a compressed feature vector with extremely low dimension, the dimension of which is one-sixteenth or less of the original feature dimension.

7. The elastic flow splitting expert parallel acceleration method based on a global basis matrix according to claim 1, characterized in that, The step of sending the dimensionality-reduced feature vector to the target computing device via a background communication method, so that the target computing device can perform residual compensation calculation, includes: Detect the idle state or fragmented transmission window of the network link; When available bandwidth is detected on the network link, an asynchronous data packet carrying the reduced-dimensional feature vector is sent. During the intervals between completing the main computing tasks, the target computing device uses the reduced-dimensional feature vector and the pre-stored residual matrix to calculate the residual compensation amount, and then fuses the residual compensation amount with the first calculation result to restore the computing accuracy.

8. The elastic flow splitting expert parallel acceleration method based on a global basis matrix according to claim 1, characterized in that, The method is applicable to expert parallel computing scenarios between computing units within a graphics processor, between different graphics processors, or between different computing nodes.

9. The elastic flow splitting expert parallel acceleration method based on a global basis matrix according to claim 1, characterized in that, The shared basis matrix is ​​extracted from multiple original expert weight matrices by singular value decomposition or principal component analysis. The shared basis matrix has orthogonal properties and is configured to reside in the static random access memory or secondary cache of each computing device.

10. A flexible shunt expert parallel acceleration device based on a global basis matrix, characterized in that, The apparatus applied to the elastic shunt expert parallel acceleration method based on a global basis matrix according to any one of claims 1-9, the apparatus comprising: An information acquisition module is used to acquire input information to be processed and to determine the target expert of the input information and the target computing device where the target expert is located; The congestion monitoring module is used to monitor the real-time congestion status of the target computing device and determine whether a congestion condition is triggered based on the real-time congestion status. The traffic control module is used to intercept the transmission of the input information when the congestion condition is determined to be triggered, and to start the virtual expert generation process in the current computing device. The virtual expert construction module is used to extract the coefficient vector corresponding to the target expert from the locally stored set of shared basis matrices, and dynamically reconstruct the virtual expert weight matrix using the shared basis matrices and the coefficient vector; A local computing module is used to perform local forward computation on the input information using the virtual expert weight matrix to generate a first computation result; An asynchronous communication module is used to pass the first calculation result to the next layer calculation operator, perform dimensionality reduction processing on the input information to generate a dimensionality-reduced feature vector, and send the dimensionality-reduced feature vector to the target computing device through a background communication method so that the target computing device can perform residual compensation calculation.

Images