A matrix multiplication approximate calculation method based on multi-hash voting mechanism

Abstract

This application relates to a matrix multiplication approximation calculation method based on a multi-hash voting mechanism. The method includes: a source computing node in the inference system obtains the token activation value matrix of the data to be inferred, which is to be sent to the target computing node where the target expert resides; the source and target computing nodes perform hash compression and voting operations on the token activation value matrix and the expert weight matrix respectively based on multiple preset hash functions to generate corresponding sketch sets; during the All-to-All communication process at the MoE layer, the source computing node sends the sketch set of the token activation value matrix to the target computing node; the target computing node performs approximate calculations based on the rows selected from the intersection of the two sketch sets to obtain the inference result. This method improves the inference efficiency of the inference system by introducing multiple independent hash functions for collaborative sampling and voting, thereby reducing computation, memory usage, network load, and GPU idle waiting time.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology and relates to a matrix multiplication approximation calculation method based on a multi-hash voting mechanism. Background Technology

[0002] In fields such as artificial intelligence, big data analytics, and scientific computing, large-scale matrix multiplication is one of the most critical and time-consuming computational operations. With the explosive growth of model size and data volume, single-machine computing resources, memory capacity, and communication bandwidth have become bottlenecks restricting system performance. To address this, Approximate Matrix Multiplication (AMM) technology has emerged. Its core idea is to compress the original matrix (i.e., generate a sketch) and perform approximate calculations on a smaller data scale, thereby achieving significant improvements in computational, storage, and communication efficiency with a controllable loss of accuracy.

[0003] The earliest AMM method originated from the priority sampling technique in streaming algorithms. This method requires calculating precise sampling probabilities and typically requires simultaneous access to or knowledge of two matrices to be multiplied. and However, in scenarios where the matrix is ​​distributed across different computing nodes (distributed computing) or needs to be processed sequentially (streaming computing), this requirement for "simultaneous access" cannot be met, greatly limiting its application scope.

[0004] Subsequent developments in single-hash-based coordinated sampling AMM techniques have overcome the "simultaneous access" problem, but this method relies on a "single hash decision," which leads to the following prominent issues: Sampling randomness leads to information loss: a single hash function may miss "high-leverage" rows that contribute significantly to the product result or mistakenly select noisy rows that contribute little due to randomness. In order to keep the approximation error within an acceptable range, existing techniques are forced to use conservative, large-size sketches (i.e., selecting more rows), which directly undermines the compression and speedup benefits that AMM techniques aim to achieve.

[0005] Communication overhead increases linearly with sketch size: In scenarios such as distributed large model training or edge computing, matrices or their sketches need to be frequently transmitted between network nodes. Increasing the sketch size linearly amplifies network communication volume, making communication latency and bandwidth consumption key bottlenecks for overall system performance.

[0006] The high computational complexity of subsequent operations stems from the fact that approximate calculations must be performed on the intersection of the selected rows in the two sketches. Larger sketches imply a larger intersection, making it difficult to reduce the computational complexity of subsequent matrix multiplications and limiting the improvement in end-to-end computational speed.

[0007] The aforementioned shortcomings are particularly pronounced in the inference of Mixture-of-Experts (MoE) models. While MoE significantly expands model capacity by dividing a large model into multiple expert networks and introducing a dynamic routing mechanism, the forward propagation of each layer of MoE requires a global all-to-all communication to send the activation values ​​of different tokens to the corresponding expert's node. As the number of experts and the scale of nodes increase, the data volume of a single communication can reach gigabytes, far exceeding the bandwidth limit of high-speed interconnected networks. This results in communication time accounting for 30% to 50% of the total training time, forming the so-called "communication wall" problem. If existing single-hash sketch methods are used to compress and transmit data, a larger sketch size is required to ensure approximate accuracy. This, in turn, prevents sufficient reduction in communication volume and may even increase the computational burden on the receiving end due to excessive computational intersection, ultimately leading to a dilemma where "accuracy and efficiency" are difficult to balance.

[0008] In summary, existing single-hash-based coordinated sampling AMM technology cannot effectively compress sketch size while ensuring approximate accuracy. Therefore, it is difficult to achieve the ideal goal of "high accuracy, low communication, and fast computation" simultaneously in communication-intensive and high-performance computing scenarios. Summary of the Invention

[0009] To address the problems existing in the traditional methods mentioned above, this invention proposes a matrix multiplication approximation calculation method based on a multi-hash voting mechanism. This method overcomes the difficulty of balancing accuracy, communication overhead, and computational efficiency in inference systems through the innovative mechanism of multi-hash voting.

[0010] To achieve the above objectives, the embodiments of the present invention adopt the following technical solutions: On the one hand, a matrix multiplication approximation calculation method based on a multi-hash voting mechanism is provided, including the following steps: In the inference system, the source computing node obtains the token activation value matrix of the data to be inferred from the target computing node where the target expert is located in the inference system.

[0011] The source computing node performs hash compression on the token activation value matrix based on multiple preset hash functions, and uses a voting mechanism to select rows to be included in the sketch set based on the hash processing results, thereby generating a sketch set of the token activation value matrix.

[0012] During the All-to-All communication process at the MoE layer, the source compute node sends a sketch set of the token activation value matrix to the target compute node.

[0013] The target computing node performs hash compression processing on the locally stored expert weight matrix based on multiple preset hash functions, and uses a voting mechanism to select the rows to be included in the sketch set based on the hash processing results, thereby generating a sketch set of the expert weight matrix.

[0014] During the forward computation of the expert network in the MoE layer, the target computation node performs approximate computation based on the row selected by the intersection of the sketch sets of the token activation value matrix and the expert weight matrix, and obtains the approximate output result of the expert network as the inference result.

[0015] In one embodiment, the preset multiple hash functions are generated by c different random seeds to form c independent hash functions; the hash function is used to map any row index i of the matrix to a random number uniformly distributed in the interval [0,1], where c is an integer greater than 1.

[0016] In one embodiment, the source computing node performs hash compression processing on the token activation value matrix based on multiple preset hash functions, and uses a voting mechanism to select rows to be included in the sketch set based on the hash processing results, generating a sketch set of the token activation value matrix, including: On the source computing node: Calculate the hash value of each row of the token activation value matrix based on c hash functions; calculate the score for each hash function based on all hash values ​​in each row; determine the dynamic passing score of the token activation value matrix for each hash function based on the (k+1)th smallest score calculated for each hash function and the number of hash functions; for each row of the token activation value matrix, count the number of hash functions whose scores are less than the dynamic passing score of the token activation value matrix. When the number reaches a preset voting threshold, the corresponding row is selected into the sketch set. Record the row number corresponding to the row. .

[0017] In one embodiment, the score under each hash function is: , ,...,

[0018] in, The index of the i-th row of the token activation value matrix is ​​the score under the a-th hash function. This is the hash value obtained by processing the i-th row index of the token activation value matrix using the a-th hash function. Let be the square of the norm of the i-th row of the first matrix. , .

[0019] In one embodiment, the dynamic passing grade for the token activation value matrix is:

[0020] in, The dynamic passing grade for the token activation value matrix. Calculate the (k+1)th smallest score and the number of hash functions for the j-th hash function.

[0021] In one embodiment, the process of generating the sketch set of the expert weight matrix is ​​the same as the process of generating the sketch set of the token activation value matrix.

[0022] In one embodiment, during the forward computation of the expert network in the MoE layer, the target computation node performs approximate computation based on the rows selected by the intersection of the sketch sets of the token activation value matrix and the expert weight matrix, to obtain the approximate output of the expert network, including: During the forward computation of the expert network in the MoE layer, the target computation node takes the intersection of the sketch sets of the token activation value matrix and the expert weight matrix as the row in both the token activation value matrix and the expert weight matrix that is selected by the sketch sets of both matrices.

[0023] The approximate output of the expert network is obtained by approximating the rows selected by the sketches of both the token activation matrix and the expert weight matrix.

[0024] in, This is an approximation of the token activation value matrix and the expert weight matrix, where I is the set of row indices selected by the sketches of both the token activation value matrix and the expert weight matrix. For the i-th row of the token activation value matrix, This represents the i-th row of the expert weight matrix, where the superscript T is the transpose of the row vector. The dynamic passing grade for the token activation value matrix. This is the dynamic passing score for the expert weight matrix.

[0025] One of the above technical solutions has the following advantages and beneficial effects: The aforementioned matrix multiplication approximation calculation method based on a multi-hash voting mechanism includes: a source computing node in the inference system obtains a token activation value matrix of the data to be inferred, which is to be sent to the target computing node where the target expert resides in the inference system; the source computing node performs hash compression processing on the token activation value matrix based on multiple preset hash functions, and selects rows to be included in the sketch set using a voting mechanism on the hash processing result, generating a sketch set of the token activation value matrix; during the All-to-All communication process of the MoE layer, the source computing node sends the sketch set of the token activation value matrix to the target computing node; the target computing node processes the locally stored expert weight matrix to generate a sketch set of the expert weight matrix; during the forward computation process of the expert network in the MoE layer, approximate calculation is performed on the rows selected by the intersection of the sketch sets of the token activation value matrix and the expert weight matrix to obtain the approximate output result of the expert network, which is used as the inference result. This method introduces multiple independent hash functions for collaborative sampling and voting, which maintains or even improves the approximate accuracy at a smaller sketch size, while reducing computation, memory usage, network load, and GPU idle time, thereby improving the inference efficiency of the inference system. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of this application or the conventional technology, the drawings used in the description of the embodiments or the conventional technology will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a flowchart illustrating a matrix multiplication approximation calculation method based on a multi-hash voting mechanism in one embodiment. Figure 2 This is a schematic diagram of the matrix multiplication approximation calculation process based on a multi-hash voting mechanism in one embodiment. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0029] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.

[0030] It should be noted that, in this document, the reference to "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The presentation of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will understand that the embodiments described herein can be combined with other embodiments. The term "and / or" as used herein refers to any combination of one or more of the associated listed items, and all possible combinations, including such combinations.

[0031] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0032] In one embodiment, such as Figure 1 , Figure 2 As shown, a matrix multiplication approximation calculation method based on a multi-hash voting mechanism is provided, which may include the following processing steps 1 to 5: Step 1: The source computing node in the inference system obtains the token activation value matrix of the data to be inferred from the target computing node where the target expert is located in the inference system.

[0033] Specifically, the token activation value matrix is ​​an n-row, d-column matrix A.

[0034] The inference system can be, but is not limited to, a distributed inference system based on the MoE model, which enables multi-device collaborative inference by distributing expert networks across different computing devices to accommodate massive parameters and using gating networks for dynamic token routing.

[0035] A distributed inference system based on the MoE model refers to a hardware and software system that deploys a large language model with sparse activation characteristics (such as an expert network distributed across multiple GPUs) on multiple computing devices through dynamic routing and expert parallelism strategies to collaboratively complete text generation or question answering tasks. The core challenge of this system is the all-to-all communication bottleneck caused by cross-GPU token routing.

[0036] Step 2: The source computing node performs hash compression on the token activation value matrix based on multiple preset hash functions, and uses a voting mechanism to select rows to be included in the sketch set based on the hash processing results, thereby generating a sketch set of the token activation value matrix.

[0037] Specifically, the pre-defined multiple hash functions include c independent hash functions; the c independent hash functions are the basis for parallel operation, and their independence ensures the diversity of evaluation.

[0038] The fundamental problems of single-hash coordination sampling methods, such as the prominent contradiction between accuracy and sketch size, high communication overhead, and limited computational efficiency, are addressed by introducing a multi-hash voting mechanism. This multi-hash voting mechanism significantly enhances the robustness of the proposed method. A high-influence row, even if it scores high in a single calculation due to the accidental influence of a certain hash function, can still be selected as long as a sufficient number (at least m) of other hash functions recognize it (even if its score is low). Conversely, a low-influence "noisy row" is unlikely to gain widespread recognition from multiple independent hash functions simultaneously. This is equivalent to an efficient noise filtering and signal enhancement process.

[0039] In large-scale distributed training or inference systems based on MoE (Mixed Experts), data parallelism and expert parallelism coexist: the Gate layer routes different tokens to different experts; the routing results determine which tokens need to be sent to experts on other GPUs for computation, triggering all-to-all communication, where each GPU needs to send its tokens to all other GPUs. As the model size increases (number of experts increases) or the sequence length increases, the data transmission volume becomes enormous, severely impacting the bandwidth of NVLink or IB (InfiniBand) networks. Therefore, this application uses consensus voting through multiple independent hash functions to more accurately identify and retain key lines and filter noisy lines, resulting in a significantly smaller final effective sketch size compared to traditional methods under the same sampling budget. This dramatic reduction in sketch size leads to an order-of-magnitude decrease in the amount of data that needs to be transmitted between networks in distributed computing, directly solving the core communication bottleneck problem of high network latency and large bandwidth consumption caused by excessively large sketches.

[0040] While compressing the sketch, the accuracy of the approximate calculation is significantly improved. The multi-hash voting mechanism effectively suppresses the error caused by random collisions in a single hash, and increases the probability of important rows being stably selected through collective decision-making. Experiments show that this method achieves lower relative error in multiple tasks.

[0041] In distributed inference systems based on the MoE model, the massive amount of data generated by all-to-all communication is a technical bottleneck restricting system scalability. This invention addresses this by designing a specific multi-hash voting mechanism to compress the specific technical data—the token activation value—during transmission, thereby reducing network communication load and enabling more efficient distributed inference.

[0042] Step 3: During the All-to-All communication process at the MoE layer, the source compute node sends a sketch set of the token activation value matrix to the target compute node.

[0043] Step 4: The target computing node performs hash compression processing on the locally stored expert weight matrix based on multiple preset hash functions, and uses a voting mechanism to select the rows to be included in the sketch set based on the hash processing results, thereby generating a sketch set of the expert weight matrix.

[0044] Specifically, the expert weight matrix is ​​an n x t matrix B.

[0045] Step 5: During the forward computation of the expert network in the MoE layer, the target computation node performs approximate computation based on the row selected by the intersection of the sketch sets of the token activation value matrix and the expert weight matrix, and obtains the approximate output result of the expert network as the inference result.

[0046] Specifically, only rows selected by the sketch sets of both matrices will participate in the final approximation calculation. This is key to coordinated sampling, ensuring the alignment of product terms.

[0047] The approximation calculation utilizes rows from both the token activation matrix and the expert weight matrix that are simultaneously selected by the sketches of both matrices, significantly reducing computation time and achieving end-to-end computational acceleration. Because the size of the intersection of the rows involved in the final calculation is drastically reduced, the core computational cost of matrix multiplication is also sharply decreased. Although additional hash computation overhead is introduced, this linear overhead is far less than the gains from the computational cost reduction. While maintaining essentially the same computational accuracy, this method reduces the high computational cost and large memory usage associated with directly using a combination of columns from the token activation matrix and rows from the expert weight matrix for approximation calculations, while simultaneously improving computational speed. In practical computing scenarios, the hash computation overhead is less than 10% of the total computational cost, while this method actually reduces the total computational cost by more than half.

[0048] This solution is particularly suitable for cross-machine GPU clusters connected via InfiniBand or high-speed Ethernet, where network bandwidth (e.g., 400Gbps) is much lower than the intra-node memory bandwidth (e.g., NVLink), so the benefits of communication compression are particularly significant.

[0049] The aforementioned matrix multiplication approximation calculation method based on a multi-hash voting mechanism includes: a source computing node in the inference system obtains a token activation value matrix of the data to be inferred, which is to be sent to the target computing node where the target expert resides in the inference system; the source computing node performs hash compression processing on the token activation value matrix based on multiple preset hash functions, and selects rows to be included in the sketch set using a voting mechanism on the hash processing result, generating a sketch set of the token activation value matrix; during the All-to-All communication process of the MoE layer, the source computing node sends the sketch set of the token activation value matrix to the target computing node; the target computing node processes the locally stored expert weight matrix to generate a sketch set of the expert weight matrix; during the forward computation process of the expert network in the MoE layer, approximate calculation is performed on the rows selected by the intersection of the sketch sets of the token activation value matrix and the expert weight matrix to obtain the approximate output result of the expert network, which is used as the inference result. This method introduces multiple independent hash functions for collaborative sampling and voting, which maintains or even improves the approximate accuracy at a smaller sketch size, while reducing computation, memory usage, network load, and GPU idle time, thereby improving the inference efficiency of the inference system.

[0050] In one embodiment, the multiple hash functions preset in step 1 are generated by c different random seeds to form c independent hash functions; the hash function is used to map any row index i of the matrix to a random number uniformly distributed in the interval [0,1], where c is an integer greater than 1.

[0051] Specifically, c hash functions are generated using c different random seeds. Each hash function This maps any row index i to a random number uniformly distributed within the interval [0,1]. These c hash functions form the basis for parallel operation, and their independence ensures diversity in evaluation. In a practical implementation, they can be quickly generated using c different random number seeds.

[0052] In one embodiment, step 2 includes: on the source computing node: calculating the hash value of each row of the token activation value matrix according to c hash functions; calculating the score under each hash function based on all hash values ​​in each row; determining the dynamic passing score of the token activation value matrix for each hash function based on the (k+1)th smallest score calculated under each hash function and the number of hash functions; for each row of the token activation value matrix, counting the number of hash functions whose scores under each hash function are less than the dynamic passing score of the token activation value matrix, and when the number reaches a preset voting threshold, selecting the corresponding row into the sketch set. Record the row number corresponding to the row. .

[0053] Specifically, this method provides users with an intuitive control dimension through two core parameters: the number of hash functions *c* and the voting threshold *m*. By adjusting the configuration of (c, m), a flexible trade-off can be made between sketch size and computational accuracy, thus adapting to the specific constraints of various application scenarios, from edge computing with extremely limited communication to server-side inference with stringent accuracy requirements.

[0054] In one embodiment, the score under each hash function is: , ,...,

[0055] in, The index of the i-th row of the token activation value matrix is ​​the score under the a-th hash function. This is the hash value obtained by processing the i-th row index of the token activation value matrix using the a-th hash function. Let be the square of the norm of the i-th row of the first matrix. , .

[0056] In one embodiment, the dynamic passing grade for the token activation value matrix is:

[0057] in, The dynamic passing grade for the token activation value matrix. Calculate the (k+1)th smallest score and the number of hash functions for the j-th hash function.

[0058] Specifically, the steps for determining the sketch set of the token activation value matrix and the expert weight matrix include: Step S1: For each row A of the token activation value matrix i Calculate the score for each hash function, specifically: , ,..., .

[0059] Step S2: Describe the hash function The (k+1)th smallest fraction calculated below is ,remember The goal of this step is to determine a dynamic "passing grade" for each hash function.

[0060] Step S3: For each row A i Statistical satisfaction The number of hash functions. If this number reaches a preset voting threshold m, then the row is selected into the sketch set. The row number of this row is recorded. .in Let j be the score of the j-th row under the i-th hash function. This is the set of row numbers corresponding to each row in the sketch set used to record the token activation value matrix.

[0061] Step 4: Replace the token activation value matrix with the expert weight matrix, and repeat steps S1 to S3 to obtain... , , ,in , , These are the dynamic passing line of the expert weight matrix, the sketch set, and the set of row numbers corresponding to each row in the sketch set used to record the expert weight matrix.

[0062] In one embodiment, the process of generating the sketch set of the expert weight matrix in step 4 is the same as the process of generating the sketch set of the token activation value matrix.

[0063] In one embodiment, step 5 includes: during the forward computation of the expert network in the MoE layer, the target computation node takes the intersection of the sketch sets of the token activation value matrix and the expert weight matrix as the row selected by both the sketch sets of the token activation value matrix and the expert weight matrix; based on the row selected by both the sketch sets of the token activation value matrix and the expert weight matrix, approximate computation is performed to obtain the approximate output result of the expert network as follows:

[0064] in, This is an approximation of the token activation value matrix and the expert weight matrix, where I is the set of row indices selected by the sketches of both the token activation value matrix and the expert weight matrix. For the i-th row of the token activation value matrix, This represents the i-th row of the expert weight matrix, where the superscript T is the transpose of the row vector. The dynamic passing grade for the token activation value matrix. This is the dynamic passing score for the expert weight matrix.

[0065] It should be understood that, although the above Figure 1 The steps are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise explicitly stated in this document, there is no strict order in which these steps are executed; they can be performed in other orders. Furthermore, the above...Figure 1 At least some of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0066] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0067] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of protection of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and all such modifications and improvements fall within the scope of protection of this application.

Claims

1. A matrix multiplication approximation calculation method based on a multi-hash voting mechanism, characterized in that, Including the following steps: In the inference system, the source computing node obtains the token activation value matrix of the data to be inferred, which is to be sent to the target computing node where the target expert is located in the inference system. The source computing node performs hash compression processing on the token activation value matrix based on multiple preset hash functions, and uses a voting mechanism to select rows to be included in the sketch set based on the hash processing results, thereby generating a sketch set of the token activation value matrix. During the All-to-All communication process at the MoE layer, the source computing node sends a set of sketches of the token activation value matrix to the target computing node; The target computing node performs hash compression processing on the locally stored expert weight matrix based on multiple preset hash functions, and uses a voting mechanism to select the rows to be included in the sketch set based on the hash processing results, thereby generating a sketch set of the expert weight matrix. During the forward computation of the expert network in the MoE layer, the target computation node performs approximate computation based on the row selected by the intersection of the sketch sets of the token activation value matrix and the expert weight matrix, and obtains the approximate output result of the expert network as the inference result.

2. The matrix multiplication approximation calculation method based on multi-hash voting mechanism according to claim 1, characterized in that, The preset multiple hash functions are generated by c different random seeds to form c independent hash functions; the hash function is used to map any row index i of the matrix to a random number uniformly distributed in the interval [0,1], where c is an integer greater than 1.

3. The matrix multiplication approximation calculation method based on multi-hash voting mechanism according to claim 2, characterized in that, The source computing node performs hash compression processing on the token activation value matrix based on multiple preset hash functions. A voting mechanism is then used to select rows for inclusion in the draft set of the hash processing results, generating a draft set of the token activation value matrix, including: On the source computing node: The hash value of each row of the token activation value matrix is ​​calculated based on c hash functions; Calculate the score for each hash function based on all hash values ​​in each row; Based on the (k+1)th smallest score calculated under each hash function and the number of hash functions, determine the dynamic passing score of the token activation value matrix for each hash function; For each row of the token activation value matrix, count the number of hash functions whose scores under each hash function are less than the dynamic passing line of the token activation value matrix. When the number reaches a preset voting threshold, the corresponding row is selected into the sketch set. Record the row number corresponding to the row. .

4. The matrix multiplication approximation calculation method based on multi-hash voting mechanism according to claim 3, characterized in that, The score for each hash function is: ， ，...， in, The index of the i-th row of the token activation value matrix is ​​the score under the a-th hash function. This is the hash value obtained by processing the i-th row index of the token activation value matrix using the a-th hash function. Let be the square of the norm of the i-th row of the first matrix. , .

5. The matrix multiplication approximation calculation method based on multi-hash voting mechanism according to claim 3, characterized in that, The dynamic passing grade for the token activation value matrix is: in, The dynamic passing grade for the token activation value matrix. Calculate the (k+1)th smallest score and the number of hash functions for the j-th hash function.

6. The matrix multiplication approximation calculation method based on multi-hash voting mechanism according to claim 3, characterized in that, The process of generating the sketch set of the expert weight matrix is ​​the same as the process of generating the sketch set of the token activation value matrix.

7. The matrix multiplication approximation calculation method based on multi-hash voting mechanism according to claim 1, characterized in that, During the forward computation of the expert network in the MoE layer, the target computation node performs approximate computation based on the row selected by the intersection of the sketch sets of the token activation value matrix and the expert weight matrix, to obtain the approximate output result of the expert network, including: During the forward computation of the expert network in the MoE layer, the target computation node takes the intersection of the sketch sets of the token activation value matrix and the expert weight matrix as the row in the token activation value matrix and the expert weight matrix that is selected by the sketch sets of both matrices. The approximate output of the expert network is obtained by approximating the rows selected by the sketches of both the token activation matrix and the expert weight matrix. in, This is an approximation of the token activation value matrix and the expert weight matrix, where I is the set of row indices selected by the sketches of both the token activation value matrix and the expert weight matrix. For the i-th row of the token activation value matrix, This represents the i-th row of the expert weight matrix, where the superscript T is the transpose of the row vector. The dynamic passing grade for the token activation value matrix. This is the dynamic passing score for the expert weight matrix.

Images