Acceleration method and related equipment for performing computational tasks using a mixture of experts model

Abstract

The present application belongs to the field of artificial intelligence chip acceleration technology, and proposes an acceleration method and related equipment for an expert mixture model to perform computing tasks. The method includes: obtaining the computing task to be processed, and determining the target expert network in the expert mixture model for performing the computing task to be processed, the network parameters of each first expert network in the expert mixture model are stored in the on-chip memory of the accelerator chip, and the network parameters of each second expert network in the expert mixture model are stored in the off-chip memory of the accelerator chip; calling the first network parameters of the first expert network in the target expert network from the on-chip memory, and calling the second network parameters of the second expert network in the target expert network from the off-chip memory; based on the first network parameters and the second network parameters, executing the computing task to be processed through the target expert network in the expert mixture model. The technical solution provided by the present application can improve the acceleration efficiency of the expert mixture model in performing computing tasks.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence chip acceleration, and in particular relates to an acceleration method and related equipment for performing computing tasks using an expert mixture model. Background Art

[0002] With the rapid development of technologies such as artificial intelligence and deep learning, the scale and complexity of neural network models continue to increase, and the computing resource consumption during model inference and training is also increasing. To meet the needs of high-performance computing, various specialized accelerator chips are widely used in the inference and training of deep neural network models. At the same time, the Mixture of Experts (MoE) model, as an innovative model structure that can improve model capacity and expressiveness while reducing computing resource consumption, has gradually become a focus of attention in industry and academia. The MoE model introduces multiple sub-networks (i.e., expert networks) and dynamically selects some expert networks for inference based on the characteristics of the input task, thereby achieving on-demand allocation and efficient utilization of computing resources.

[0003] However, as the number of expert networks increases, the total number of network parameters in the mixture of expert models also expands dramatically, resulting in an increasing amount of memory space occupied by accelerator chips, which in turn restricts the efficiency of accelerator chips. This has become a key bottleneck restricting the practical deployment of the mixture of expert models and their ability to accelerate computational tasks. Therefore, improving the efficiency of the mixture of expert models in executing computational tasks is a pressing technical issue. Summary of the Invention

[0004] The embodiments of the present application provide a method, device, computer program product, computer-readable storage medium, and electronic device for accelerating an expert mixture model to perform computing tasks, thereby improving the acceleration efficiency of the expert mixture model to perform computing tasks to a certain extent.

[0005] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by practice of the present application.

[0006] According to a first aspect of an embodiment of the present application, a method for accelerating the execution of a computing task by an expert mixture model is provided, wherein the expert mixture model is deployed in an accelerator chip, and the method includes: obtaining a computing task to be processed, and determining a target expert network in the expert mixture model for executing the computing task to be processed, wherein the network parameters of each first expert network in the expert mixture model are stored in an on-chip memory of the accelerator chip, and the network parameters of each second expert network in the expert mixture model are stored in an off-chip memory of the accelerator chip; calling the first network parameters of the first expert network in the target expert network from the on-chip memory, and calling the second network parameters of the second expert network in the target expert network from the off-chip memory; and executing the computing task to be processed through the target expert network in the expert mixture model based on the first network parameters and the second network parameters.

[0007] In some embodiments of the present application, based on the aforementioned scheme, determining the target expert network in the expert mixture model for executing the computational task to be processed includes: calculating in parallel in the accelerator chip the similarity between the computational task to be processed and each expert network in the expert mixture model; and selecting a set number of expert networks with the highest similarity as the target expert networks for executing the computational task to be processed.

[0008] In some embodiments of the present application, based on the aforementioned scheme, the similarity between the computing task to be processed and each expert network in the expert mixture model is calculated in parallel in the accelerator chip, including: obtaining the feature matrix of the computing task to be processed and the expert weight matrix of each expert network in the expert mixture model; based on the feature matrix and the expert weight matrix, the similarity between the computing task to be processed and each expert network in the expert mixture model is calculated in parallel in the accelerator chip.

[0009] In some embodiments of the present application, based on the aforementioned scheme, the parallel calculation of the similarity between the computing task to be processed and each expert network in the expert mixture model in the accelerator chip includes: parallel calculation of the similarity between the computing task to be processed and each expert network in the expert mixture model in the accelerator chip according to a Hamming distance algorithm, a cosine similarity algorithm, or a gated network.

[0010] In some embodiments of the present application, based on the aforementioned solution, the expert weight matrices of the respective expert networks in the mixture of experts model are pre-stored in a distributed lookup table of the accelerator chip.

[0011] In some embodiments of the present application, based on the aforementioned solution, the method further includes: updating the expert weight matrix stored in the distributed lookup table in real time based on a boundary scan test interface.

[0012] In some embodiments of the present application, based on the aforementioned scheme, the method further includes: in response to an instruction to update the network parameters in the on-chip memory, counting the frequency of each expert network in the expert mixture model performing computing tasks in history; defining the set number of expert networks with the highest frequency as the first expert network, and defining the expert networks other than the first expert network in the expert mixture model as the second expert network; saving the network parameters of the first expert network in the on-chip memory of the accelerator chip, and saving the network parameters of the second expert network in the off-chip memory of the accelerator chip.

[0013] In some embodiments of the present application, based on the aforementioned solution, the method further includes: triggering an update instruction for the network parameters in the on-chip memory before calling the first network parameters of the first expert network in the target expert network from the on-chip memory and calling the second network parameters of the second expert network in the target expert network from the off-chip memory.

[0014] In some embodiments of the present application, based on the aforementioned solution, the method further includes: abstracting the common operators in the accelerator chip into configurable modules; and running each target expert network through the configurable modules based on a time division multiplexing algorithm.

[0015] In some embodiments of the present application, based on the aforementioned solution, the accelerator chip includes any one of a field programmable gate array chip, a graphics processing chip, a Google tensor processing chip, and a dedicated integrated circuit.

[0016] According to a second aspect of an embodiment of the present application, an acceleration device for executing a computing task using an expert mixture model is provided, wherein the expert mixture model is deployed in an accelerator chip, and the device includes: an acquisition unit for acquiring the computing task to be processed and determining a target expert network in the expert mixture model for executing the computing task to be processed, wherein the network parameters of each first expert network in the expert mixture model are stored in the on-chip memory of the accelerator chip, and the network parameters of each second expert network in the expert mixture model are stored in the off-chip memory of the accelerator chip; a calling unit for calling the first network parameters of the first expert network in the target expert network from the on-chip memory, and calling the second network parameters of the second expert network in the target expert network from the off-chip memory; and an execution unit for executing the computing task to be processed through the target expert network in the expert mixture model based on the first network parameters and the second network parameters.

[0017] According to a third aspect of an embodiment of the present application, a computer program product is provided, which includes computer instructions, which are stored in a computer-readable storage medium and are suitable for being read and executed by a processor, so that a computer device having the processor executes to implement the operations performed by the method described in the first aspect above.

[0018] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, in which at least one computer program instruction is stored. The at least one computer program instruction is loaded and executed by a processor to implement the operations performed by the method described in the first aspect above.

[0019] According to a fifth aspect of an embodiment of the present application, an electronic device is provided, comprising one or more processors and one or more memories, wherein at least one computer program instruction is stored in the one or more memories, and the at least one computer program instruction is loaded and executed by the one or more processors to implement the operations performed by the method described in the first aspect above.

[0020] Based on the technical solution proposed in this application, by deploying an expert mixture model in an accelerator chip and storing the parameters of different expert networks in on-chip memory and off-chip memory respectively, the acceleration efficiency of the model in executing computing tasks can be effectively improved. Specifically, the parameters of the first expert network that are commonly used or frequently calculated can be stored in the on-chip memory, which can fully utilize the high-speed and low-latency access characteristics of the on-chip memory to achieve rapid call of key network parameters, significantly reduce data access latency, and improve overall inference and training speed. Storing the parameters of the second expert network in off-chip memory can break through the capacity limitations of on-chip memory, support the flexible deployment of more and larger-scale expert networks, and provide guarantees for model scalability and diversity. During the execution of specific computing tasks, network parameters of on-chip memory and off-chip memory can be flexibly called according to actual needs, giving full play to the advantages of the two-level storage structure. On the one hand, the high bandwidth and low power consumption characteristics of the on-chip memory ensure the efficient processing of high-frequency access network parameters, reducing data movement and access bottlenecks; on the other hand, the capacity advantage of off-chip memory supports the complex structure and large-scale parameter storage requirements of the expert mixture model. Through this hierarchical storage and efficient scheduling mechanism, it is ultimately possible to achieve efficient acceleration of expert mixture model computing tasks, improve hardware resource utilization and overall system performance, and meet the efficient reasoning and training needs of large-scale and complex expert mixture models in practical applications.

[0021] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings are incorporated into and constitute a part of the specification, illustrating embodiments consistent with the present application and, together with the specification, explaining the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and those skilled in the art can derive other drawings based on these drawings without inventive effort. In the drawings:

[0023] Figure 1 A flowchart showing a method for accelerating the execution of computing tasks by a mixture of experts model in an embodiment of the present application is shown;

[0024] Figure 2 The following is a diagram showing a hardware architecture for executing the acceleration method in an embodiment of the present application;

[0025] Figure 3 A block diagram of an acceleration device for executing computing tasks using a mixture of experts model in an embodiment of the present application is shown;

[0026] Figure 4 A schematic structural diagram of an electronic device in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0028] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner.In the following description, many specific details are provided so as to provide a full understanding of the embodiments of the present application. However, it will be appreciated by those skilled in the art that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps etc. can be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0029] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices. It should also be noted that in the accompanying drawings, certain components that do not affect the explanation of the technical solutions of this application have been omitted for clarity.

[0030] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, while others may be combined or partially combined. Therefore, the actual execution order may vary depending on the actual situation.

[0031] In the description of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, a feature specified as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, unless otherwise specified, "plurality" means two or more.

[0032] In order to enable those skilled in the art to better understand this application, the technical concepts and application background involved in this application are first briefly explained.

[0033] Mixture of Experts (MoE): The MoE is an ensemble learning method whose core concept is to combine multiple "expert models" and dynamically select or weight the outputs of different expert models based on the characteristics of the input data, thereby improving the overall model's expressiveness and generalization performance. Specifically, the MoE introduces multiple sub-networks (i.e., expert networks) and dynamically selects some expert networks for inference based on the characteristics of the input task, thereby achieving on-demand allocation and efficient utilization of computing resources. This dynamic selection mechanism enables the model to flexibly adjust the allocation of computing resources when processing different types of data, thereby improving overall performance. MoE is widely used in fields such as natural language processing and computer vision, and is particularly used in large-scale models (such as GPT-4 and SwitchTransformer) to improve parameter utilization and inference efficiency.

[0034] Accelerator Chip: An accelerator chip is an integrated circuit (chip) designed specifically for a specific computing task or application scenario. Its purpose is to outperform general-purpose processors (such as CPUs) in terms of energy consumption, speed, or efficiency. Accelerator chips typically utilize hardware-level optimizations for a specific type of computing to achieve higher parallelism, lower latency, or better energy efficiency.

[0035] In the present application, the accelerator chip may include any one of a field programmable gate array chip (FPGA), a graphics processing unit (GPU), a Google tensor processing unit (TPU), and an application-specific integrated circuit (ASIC).

[0036] With the rapid development of technologies like artificial intelligence and deep learning, the scale and complexity of neural network models are constantly increasing, and the computing resource consumption during model inference and training is also increasing. To meet the demand for high-performance computing, various specialized accelerator chips are widely used in the inference and training of deep neural network models. These accelerator chips can significantly improve the execution efficiency and response speed of models through parallel computing and optimized hardware architecture.

[0037] At the same time, the expert mixture model, as an innovative model structure that can improve model capacity and expressiveness while reducing computing resource consumption, has gradually become the focus of attention in the industry and academia.

[0038] However, as the number of expert networks increases, the total number of network parameters in the mixture of expert models also expands dramatically, resulting in an increasing demand for memory space in the accelerator chip. This increased memory usage not only affects the efficiency of the accelerator chip but also becomes a key bottleneck restricting the practical deployment of the mixture of expert models and accelerating their computational tasks. Effectively managing and optimizing these parameters becomes a significant challenge, especially in resource-constrained environments.

[0039] Currently, researchers are exploring various technical approaches to address this challenge, including but not limited to model compression, parameter sharing, and the introduction of more efficient storage management strategies. These methods aim to improve the feasibility and efficiency of expert mixture models in practical applications. In this context, this application also proposes an acceleration scheme for expert mixture models to improve their efficiency in executing computational tasks.

[0040] The following describes the implementation details of the technical solution of the embodiment of the present application:

[0041] Reference Figure 1 , shows a flow chart of an acceleration method for executing a computing task using an expert mixture model in an embodiment of the present application. The acceleration method for executing a computing task using an expert mixture model can be executed by a device having a computing processing function, wherein the expert mixture model can be deployed in an accelerator chip. Figure 1 As shown, the acceleration method for executing a computing task using the expert mixture model includes at least steps 110 to 130, which are described in detail as follows:

[0042] Reference Figure 1 In step 110, a computing task to be processed is obtained, and a target expert network in the expert mixture model for executing the computing task to be processed is determined. The network parameters of each first expert network in the expert mixture model are stored in the on-chip memory of the accelerator chip, and the network parameters of each second expert network in the expert mixture model are stored in the off-chip memory of the accelerator chip.

[0043] In this application, obtaining a pending computation task can be receiving specific task data that requires inference or training from external input or upper-level applications. For example, in a natural language processing scenario, the task can be a piece of text to be analyzed. In an image recognition scenario, it can be a picture to be classified. By extracting features from the pending computation task, a feature vector or feature matrix is ​​formed that can be used for subsequent model processing.

[0044] Specifically, in one embodiment, please refer to Figure 2 , shows a hardware architecture diagram for executing the acceleration method in an embodiment of the present application.

[0045] like Figure 2 As shown, the eigenvector or feature matrix is ​​input into the attention network model loaded on the accelerator chip in the form of a calculation task. The attention network model extracts features from the eigenvector or feature matrix to obtain a new eigenvector or feature matrix. The new eigenvector or feature matrix is ​​input into the expert mixture model loaded on the accelerator chip in the form of a calculation task, and the calculation result is obtained after processing by the expert mixture model.

[0046] In this application, the expert mixture model includes a large number of expert networks, each with its own independent set of network parameters, resulting in a large overall parameter size for the model. For different input computational tasks, the most appropriate expert network must be dynamically selected and its corresponding network parameters must be scheduled for computation. This means dynamically selecting several expert networks that are most suitable for the current computational task as the "target expert networks."

[0047] In this application, to optimize accelerator chip memory usage and improve inference efficiency, the expert networks in the mixture of experts model are divided into two categories: a first expert network and a second expert network. The network parameters of the first expert network can be stored in on-chip memory on the accelerator chip, such as static random access memory (SRAM) or block random access memory (BRAM). The network parameters of the second expert network are stored in off-chip memory outside the accelerator chip, such as double data rate synchronous dynamic random access memory (DDR), high bandwidth memory (HBM), dynamic random access memory (DRAM), static random access memory (SRAM), or flash memory.

[0048] In this application, using HBM instead of DDR storage can achieve a 5x bandwidth increase, but the cost increases by 30%. To balance cost and performance, a hybrid cache hierarchy can be adopted, combining BRAM (storage high-frequency experts) with HBM (storage low-frequency experts).

[0049] In the present application, the determination of the target expert network in the expert mixture model for executing the computational task to be processed may be performed according to the following steps 111 to 112:

[0050] Step 111 : calculating in parallel in the accelerator chip the similarity between the computing task to be processed and each expert network in the mixture of experts model.

[0051] Step 112 : Selecting a set number of expert networks with the highest similarity as target expert networks for executing the computational task to be processed.

[0052] In this application, the similarity between the computational task to be processed and each expert network in the expert mixture model is an indicator for measuring the "degree of adaptation" between the current computational task to be processed and each expert network, and can be calculated based on the characteristics of the input computational task and the capabilities or historical performance of the expert network.

[0053] Specifically, in the present application, the similarity between the to-be-processed computing task and each expert network in the expert mixture model is calculated in parallel in the accelerator chip, and the calculation can be performed according to the following steps 1111 to 1112:

[0054] Step 1111 , obtaining the feature matrix of the computation task to be processed and the expert weight matrix of each expert network in the expert mixture model.

[0055] Step 1112 : Based on the feature matrix and the expert weight matrix, similarities between the processing task to be processed and each expert network in the expert mixture model are calculated in parallel in the accelerator chip.

[0056] In this application, the feature matrix is ​​a vector or matrix expression formed after feature extraction of the current computing task to be processed (such as input samples, requests, data blocks, etc.). It can be a real-valued vector obtained by embedding (embedding), encoding (encoding), etc. the original data of the computing task to be processed, or it can be an intermediate representation obtained through a preprocessing network (such as convolution, transformer, MLP, etc.).

[0057] In this application, the expert weight matrix is ​​a set of parameters used by each expert network to characterize its "expertise direction" or "capability space", usually a feature vector or parameter vector for each expert. It can be a partial weight of the expert network (such as the weight vector of the last layer) or an "expert description vector" specifically designed for similarity calculation.

[0058] In this application, the expert weight matrices for each expert network in the mixture of experts model can be pre-stored in a distributed lookup table (LUT) on the accelerator chip. Accelerator chips (such as FPGAs) contain a large number of programmable LUT resources, which can both implement logic functions and serve as efficient on-chip storage. Through a distributed approach, each LUT can independently store the expert weight matrix for an expert network. For example, if there are 128 expert networks, these 128 expert weight matrices can be stored in 128 separate LUTs, achieving physical isolation and parallel access. When calculating the similarity between the pending computation task and the individual expert networks in the mixture of experts model, the weight matrices for the 128 expert networks can be read simultaneously and independently from their respective LUTs at high speed, eliminating serial wait times and significantly improving computational efficiency. Therefore, using the accelerator chip's distributed lookup table to store expert weight matrices can significantly reduce memory access latency, support high-frequency, low-latency expert network selection requests, and achieve microsecond-level expert network selection decisions, making it suitable for large-scale inference or online real-time decision-making scenarios.

[0059] In this application, the accelerator chip features a massively parallel processing unit, which simultaneously computes similarities between the feature matrix of the task being processed and the expert weight matrices of each expert network in the expert mixture model. This allows the similarity between the task being processed and all expert networks to be outputted simultaneously, improving computational speed. Traditional solutions typically rely on the CPU to select the expert network. However, this results in hardware resource fragmentation and the separation of the computing unit from the selection logic, leading to decision delays exceeding 50μs. This is far from meeting the computational requirements of real-time edge device scenarios (such as autonomous driving, which requires real-time computing).

[0060] In the present application, the expert weight matrix stored in the distributed lookup table can be updated in real time based on the boundary scan test interface.

[0061] In this application, the expert weight matrices of each expert network in the expert mixture model can be flexibly, online, and seamlessly upgraded, maintained, and dynamically adjusted based on requirements such as model optimization, online learning, and model upgrades. If the weight matrix is ​​hard-coded inside the accelerator chip, it will not be able to flexibly adapt to model changes. Therefore, this application can update the expert weight matrix stored in the distributed lookup table in real time based on the boundary scan test interface.

[0062] Specifically, the Boundary Scan (JTAG) interface is a chip-level debugging and data access interface used to connect the hardware devices between the accelerator chip and the CPU. Through this interface, the expert weight matrix stored in the distributed lookup table (LUT) can be accessed and modified in real time while the accelerator chip is running.

[0063] In this application, the operational process for updating the expert weight matrix in the distributed lookup table can be as follows: first, an external master control (such as a CPU) establishes communication with the accelerator chip through a boundary scan test interface. Then, through JTAG instructions, the new expert weight matrix is ​​written one by one or in batches into the LUT specified by the accelerator chip. The LUT content is refreshed in real time, and the expert weight matrix takes effect immediately without restarting or reconfiguring the chip. This provides a solid technical guarantee for the efficient deployment, continuous optimization, and intelligent evolution of the expert hybrid model. In addition, the boundary scan interface can also support regional, batch, and atomic writing to ensure data consistency and stability during the expert weight matrix update process.

[0064] In the present application, further, the similarity between the to-be-processed computing task and each expert network in the expert mixture model is calculated in parallel in the accelerator chip according to the following step 1113:

[0065] Step 1113 , based on a Hamming distance algorithm or a cosine similarity algorithm or a gated network, similarities between the processing task and each expert network in the mixture of experts model are calculated in parallel in the accelerator chip.

[0066] In this application, the Hamming distance algorithm is used when the feature matrix and expert weight matrix of the task to be processed are binary (0 / 1) or fixed-length discretely coded. The calculation method is to count the number of differences in the corresponding bits of the feature matrix and expert weight matrix of the task to be processed, i.e., the Hamming distance. In terms of hardware implementation, the accelerator chip can quickly calculate the Hamming distance between the task to be processed and the expert network through parallel XOR gates and adder arrays, greatly improving processing speed.

[0067] In this application, the cosine similarity algorithm can be used in scenarios where the feature matrix and expert weight matrix of the task to be processed are real numbers or high-dimensional floating-point vectors. The calculation method is to calculate the cosine value of the angle between the feature matrix of the task to be processed and the expert weight matrix vector through dot product and norm normalization. The larger the value, the higher the similarity. In hardware implementation, multiple parallel multiplication and addition units can be configured inside the accelerator chip to realize the dot product and modulus length calculation of the input vector and multiple expert weight matrices, support pipeline parallel processing, and significantly shorten the calculation delay.

[0068] In this application, the gated network is suitable for applications requiring a more complex expert selection mechanism. The calculation method is to treat the gated network as a small neural network, with the input being the feature matrix of the computational task to be processed, and the output being the selection probability or weight of each expert network. In terms of hardware implementation, a lightweight gated network inference unit is integrated into the accelerator chip, leveraging parallel computing resources to simultaneously score multiple expert networks, thereby achieving efficient expert network allocation.

[0069] In this application, by implementing the calculation method of Hamming distance, cosine similarity, or gated network similarity in parallel in the accelerator chip, high-speed similarity evaluation between the processing task and the expert network can be achieved, giving full play to the advantages of hardware parallel computing, greatly improving the efficiency and flexibility of expert selection, and providing support for high-performance reasoning of expert mixture models.

[0070] In step 112, a set number of expert networks with the highest similarity are selected as the target expert networks for executing the pending computation task. Specifically, the set number N of target expert networks can be determined based on the actual model design, hardware resources, and task requirements. For example, two, four, or eight target expert networks with the highest similarity can be selected. This parameter can be dynamically adjusted to accommodate varying task complexities or hardware load conditions. Next, after obtaining the similarity scores of all expert networks, the N expert networks with the highest scores need to be efficiently selected. This can be achieved by employing efficient algorithms such as parallel sorting, priority queues, and local maximum reduction on accelerator chips to further shorten the selection time. For example, FPGA chips can implement parallel comparison trees, GPU chips can use block reduction algorithms, and ASIC chips can customize dedicated selection circuits. Finally, the N selected expert networks become the target expert networks for executing the pending computation task. Based on the storage location of these target expert networks' network parameters (e.g., on-chip memory or off-chip memory), the corresponding network parameters can be dispatched to the computing unit to subsequently execute the pending computation task.

[0071] Continue to refer to Figure 1 In step 120, the first network parameters of the first expert network in the target expert network are called from the on-chip memory, and the second network parameters of the second expert network in the target expert network are called from the off-chip memory.

[0072] Please continue to refer to Figure 2 In this application, after determining the target expert network for executing the pending computation task, if the target expert network includes a first expert network, the first network parameters can be read directly from on-chip memory at high speed. If the target expert network includes a second expert network, the required second network parameters can be retrieved from off-chip memory via an on-chip controller (e.g., DMA or AXI bus).

[0073] In this application, the network parameters of some expert networks are stored in on-chip memory, and the network parameters of some expert networks are stored in off-chip memory. The advantage is that it can reduce the amount of memory space occupied in the accelerator chip, thereby effectively alleviating the on-chip memory pressure caused by the increase in the number of expert networks and achieving efficient use of memory resources. Furthermore, by releasing the memory resources of the accelerator chip, memory resources can be allocated to more computing units, thereby improving the overall parallel computing capability and task processing throughput of the accelerator chip. In addition, the reduction in on-chip memory pressure can also help reduce memory access conflicts and bandwidth bottlenecks, optimize data flow paths, and improve data processing efficiency. At the same time, it also provides greater flexibility and scalability for subsequent model upgrades, expert network expansion or multi-model collaborative deployment, which helps to meet diverse application needs.

[0074] Furthermore, in the present application, the first expert network may be an expert network that is used more frequently, and the second expert network may be an expert network that is used less frequently. That is, the parameters of the expert networks that are called more frequently are stored in the on-chip memory first, which can significantly improve the access speed of these expert networks during reasoning or training, reduce parameter loading delays, and ensure the real-time and high efficiency of high-frequency tasks. The parameters of the expert networks that are called less frequently are stored in off-chip memory and loaded on demand when needed, which can avoid the waste of on-chip memory resources. Through the hierarchical storage mechanism, not only can the accelerator chip's dependence on large-capacity on-chip memory be reduced, reducing hardware costs, but also the efficient deployment and expansion of the expert mixture model can be achieved, and the acceleration efficiency of the expert mixture model in executing computing tasks can be improved.

[0075] In this application, the following steps 113 to 115 may also be performed:

[0076] Step 113 : In response to the network parameter update instruction in the on-chip memory, counting the frequency of each expert network in the expert mixture model performing the computing task in history.

[0077] Step 114 : defining the set number of expert networks with the highest frequencies as the first expert network, and defining the expert networks other than the first expert network in the expert mixture model as the second expert network.

[0078] Step 115 : Save the network parameters of the first expert network in the on-chip memory of the accelerator chip, and save the network parameters of the second expert network in the off-chip memory of the accelerator chip.

[0079] In this application, after receiving an update instruction for network parameters in the on-chip memory, the historical execution frequency of computational tasks for each expert network in the expert mixture model can be counted. Specifically, a counter can be used to record the number of times each expert network is scheduled, called, or participates in inference / training tasks, forming a statistical table of expert network access frequencies. This statistical process can be dynamically updated based on a fixed time window (such as the last N tasks) or a sliding window to reflect the activity level of each expert network under the current actual workload. This provides a data foundation for the subsequent hierarchical storage and resource allocation of expert networks.

[0080] After completing the frequency statistics, a set number of expert networks with the highest historical computation frequency (e.g., the top K) can be defined as the "first expert network," while the remaining expert networks are defined as the "second expert network." This division ensures that expert networks that are frequently called and have a greater impact on overall model performance receive priority attention. The set number can be flexibly adjusted based on the actual on-chip memory capacity, the scale of the expert networks, and application requirements, thereby achieving a balance between performance and resource utilization.

[0081] Based on the above division results, the network parameters of the first expert network can be preferentially stored in the on-chip memory of the accelerator chip so that they can be quickly accessed in subsequent computing tasks, achieving microsecond responses. At the same time, the number of accesses to off-chip memory can be reduced, reducing the delay in loading network parameters, and improving the response speed and execution efficiency of high-frequency tasks. The network parameters of the second expert network are stored in off-chip memory and loaded into on-chip memory only when needed for calculation. This can significantly reduce the pressure on on-chip memory, freeing up more memory resources for other key modules or computing tasks, thereby improving the utilization of on-chip memory resources and achieving the effect of trading time for space. For example, in a large-scale language model inference server, some expert networks are frequently selected to handle mainstream semantic tasks, and their parameters are resident in on-chip memory, achieving microsecond responses. Expert networks that process special domain corpora are stored in off-chip memory and dynamically loaded only when specific inputs appear. For example, in edge AI devices, on-chip memory is more limited, and hierarchical storage management is implemented for expert network parameters, which not only ensures the rapid response of high-frequency expert networks in inference tasks, but also takes into account the balance between model capacity and hardware resources, and can effectively balance model capacity and real-time requirements.

[0082] Based on the technical solution from steps 113 to 115 above, the expert hybrid model can dynamically adjust parameter storage strategies based on actual operating conditions, achieving hierarchical management with "hot experts always resident and cold experts loaded on demand." This mechanism not only improves on-chip memory utilization, preventing low-frequency expert networks from occupying valuable on-chip resources, but also ensures efficient access to high-frequency expert networks, optimizing overall system performance. Furthermore, this solution exhibits excellent adaptability and scalability, flexibly responding to changes in expert network size or workload fluctuations, meeting the dual requirements of high performance and high resource utilization in practical applications.

[0083] In this application, the following step 1131 may also be performed:

[0084] Step 1131: before calling the first network parameters of the first expert network in the target expert network from the on-chip memory and calling the second network parameters of the second expert network in the target expert network from the off-chip memory, triggering an update instruction for the network parameters in the on-chip memory.

[0085] In the present application, before calling the target expert network parameters (i.e., calling the first network parameters of the first expert network from the on-chip memory, and calling the second network parameters of the second expert network from the off-chip memory), by actively triggering the update instruction of the network parameters in the on-chip memory, it can be ensured that before each actual computing task arrives, the parameter storage and scheduling strategy can be dynamically optimized based on the latest expert network usage.

[0086] Specifically, triggering an update instruction recounts the call frequency of each expert network during recent computational tasks, and based on this, determines which expert networks are frequently used (the first expert network) and which are less frequently used (the second expert network). Based on these statistical results, the storage location of the expert network parameters can be dynamically adjusted, for example, migrating some high-frequency expert network parameters to on-chip memory and those of low-frequency expert networks to off-chip memory. The activity of different expert networks may fluctuate with changes in the computational task type, input data, or workload. By issuing an update instruction before each parameter call, the distribution of expert network parameters can be adjusted in a timely manner, avoiding waste of on-chip memory resources or bottlenecks, thereby dynamically optimizing resource utilization and system performance. For example, in scenarios such as multi-task inference and online model learning, the activity of different expert networks may vary depending on the input data or task type. By triggering an update instruction before each parameter call, the system can always use the optimal expert network parameter layout to respond to the current task, improving system responsiveness and throughput.

[0087] Continue to refer to Figure 1 In step 130, based on the first network parameters and the second network parameters, the to-be-processed computing task is executed by the target expert network in the expert mixture model.

[0088] In the present application, the first network parameters and the second network parameters can first be loaded into the computing unit, and then the input data corresponding to the to-be-processed computation task is distributed to the corresponding expert sub-network in the target expert network through the routing mechanism of the expert mixture model. Each expert sub-network independently or collaboratively completes the computation of the sub-task based on its own network parameters, and finally outputs the computation results of each expert network, and is fused or selected according to the model design to obtain the final computation result output. In the present application, the high-frequency expert network parameters are permanently stored in the on-chip memory, while the low-frequency expert network parameters are loaded from the off-chip memory on demand, which can ensure efficient parameter access and computing resource utilization, thereby ensuring that the system can efficiently respond to diverse task requirements even when hardware resources are limited. In scenarios such as large-scale multi-task reasoning and personalized model services, different tasks will activate different expert networks, and the system can flexibly and efficiently schedule the required expert networks to complete the processing of complex tasks. Therefore, by combining the first network parameters with the second network parameters, the target expert networks in the expert mixture model are driven to work collaboratively, thereby achieving efficient execution of the to-be-processed computation tasks and ensuring the flexibility, scalability and optimal utilization of system resources of the model.

[0089] In this application, the following steps 131 to 132 may also be performed:

[0090] Step 131: abstract the common operators in the accelerator chip into configurable modules.

[0091] Step 132: Run each target expert network through the configurable module based on a time division multiplexing algorithm.

[0092] In this application, common operators in deep learning models (such as the GELU activation function, Layer-Norm normalization layer, convolutional layer, and fully connected layer) can be abstracted and designed as configurable modules. These modules have flexible functional and parameter interfaces and can be configured and reused according to the actual needs of different expert networks. This significantly reduces hardware resource redundancy, improves chip utilization, and facilitates the adaptation and expansion of subsequent diversified models. It has been verified that by using time division multiplexing (TDM) to serve 32 expert networks, the resource utilization of a single GELU activation function can be increased by 4 times.

[0093] Furthermore, the configurable modules can be dynamically scheduled using a time-division multiplexing algorithm. Specifically, access to the configurable modules can be allocated in rotating time slices, based on the target expert network currently in operation. During each time slot, the module loads the parameters and configuration of the corresponding expert network, completes the specified computational task, and then switches to the next expert network at the end of the time slot. In this way, shared operators enable efficient reuse and dynamic allocation of limited hardware resources across multiple expert networks, while also significantly reducing memory usage.

[0094] Based on the technical solution described in steps 131 and 132, by abstracting common operators into configurable modules and combining them with a time-division multiplexing algorithm for resource scheduling, hardware resource utilization and system flexibility can be significantly improved. This not only supports high-concurrency and high-throughput operation of multi-expert networks, but also provides an efficient and scalable hardware foundation for large-scale inference and training of expert mixture models. Taking DeepSeekR1's 128-expert model as an example, the input text vector is matched by the LUT selection engine, and the top eight experts are selected. The popularity statistics module marks these eight experts as high-frequency experts and preloads their weights from DDR into BRAM. Simultaneously, a shared GELU unit services the two experts via time-division multiplexing (TDM). The final output is weighted and aggregated before being passed to the next layer. The latency of the entire process has been verified to be within 8μs.

[0095] Based on the technical solution proposed in this application, by deploying an expert mixture model in an accelerator chip and storing the parameters of different expert networks in on-chip memory and off-chip memory respectively, the acceleration efficiency of the model in executing computing tasks can be effectively improved. Specifically, the parameters of the first expert network that are commonly used or frequently calculated can be stored in the on-chip memory, which can fully utilize the high-speed and low-latency access characteristics of the on-chip memory to achieve rapid call of key network parameters, significantly reduce data access latency, and improve overall inference and training speed. Storing the parameters of the second expert network in off-chip memory can break through the capacity limitations of on-chip memory, support the flexible deployment of more and larger-scale expert networks, and provide guarantees for model scalability and diversity. During the execution of specific computing tasks, network parameters of on-chip memory and off-chip memory can be flexibly called according to actual needs, giving full play to the advantages of the two-level storage structure. On the one hand, the high bandwidth and low power consumption characteristics of the on-chip memory ensure the efficient processing of high-frequency access network parameters, reducing data movement and access bottlenecks; on the other hand, the capacity advantage of off-chip memory supports the complex structure and large-scale parameter storage requirements of the expert mixture model. Through this hierarchical storage and efficient scheduling mechanism, it is ultimately possible to achieve efficient acceleration of expert mixture model computing tasks, improve hardware resource utilization and overall system performance, and meet the efficient reasoning and training needs of large-scale and complex expert mixture models in practical applications.

[0096] The accelerator chip computing task acceleration method based on the expert mixture model proposed in this application can be widely applied to the following typical application scenarios:

[0097] 1. Large-scale intelligent inference platforms. Scenarios such as cloud-based AI inference servers and data centers require efficient processing of massive concurrent inference requests, placing high demands on system throughput, latency, and energy efficiency.

[0098] 2. Intelligent edge devices. Embedded devices such as smart cameras, drones, self-driving cars, and smart homes are limited by hardware resources and power consumption, and require high-performance model inference within limited chip resources.

[0099] 3. High-speed image / video processing. Scenarios such as real-time video analysis, image recognition, and object detection require dynamic selection of the optimal expert network for different input features to improve model inference accuracy and processing speed.

[0100] 4. Natural language processing and speech recognition. In areas such as machine translation, speech recognition, and dialogue systems, expert mixture models can dynamically allocate computing resources based on different text / speech features, improving system response speed and intelligence.

[0101] 5. Intelligent manufacturing and industrial automation. Fields such as industrial robots and intelligent testing equipment require efficient processing of complex and changing input tasks. Mixed-expert models can dynamically adapt to task characteristics, improving production efficiency and intelligence.

[0102] 6. Financial risk control and intelligent recommendation. In scenarios such as financial anti-fraud and intelligent recommendation systems, expert hybrid models can efficiently process diverse input features, improving the model's risk identification and personalized recommendation capabilities.

[0103] The following describes an apparatus embodiment of the present application, which can be used to implement the method for accelerating computational tasks using a mixture of experts model described in the aforementioned embodiments of the present application. The mixture of experts model is deployed in an accelerator chip. For details not disclosed in the apparatus embodiment of the present application, please refer to the aforementioned embodiment of the method for accelerating computational tasks using a mixture of experts model.

[0104] See also Figure 3 , shows a block diagram of an acceleration device for performing computing tasks using an expert mixture model in an embodiment of the present application.

[0105] like Figure 3 As shown, the acceleration device 300 for executing a computing task using a mixture of experts model according to an embodiment of the present application includes: an acquisition unit 301 , a calling unit 302 and an execution unit 303 .

[0106] Among them, the acquisition unit 301 is used to obtain the computing task to be processed and determine the target expert network in the expert mixture model for executing the computing task to be processed. The network parameters of each first expert network in the expert mixture model are stored in the on-chip memory of the accelerator chip, and the network parameters of each second expert network in the expert mixture model are stored in the off-chip memory of the accelerator chip; the calling unit 302 is used to call the first network parameters of the first expert network in the target expert network from the on-chip memory, and call the second network parameters of the second expert network in the target expert network from the off-chip memory; the execution unit 303 is used to execute the computing task to be processed through the target expert network in the expert mixture model based on the first network parameters and the second network parameters.

[0107] In some embodiments of the present application, based on the aforementioned scheme, the acquisition unit 301 is configured to: calculate in parallel in the accelerator chip the similarity between the computing task to be processed and each expert network in the expert mixture model; and select a set number of expert networks with the highest similarity as the target expert networks for executing the computing task to be processed.

[0108] In some embodiments of the present application, based on the aforementioned scheme, the acquisition unit 301 is configured to: obtain the feature matrix of the computing task to be processed and the expert weight matrix of each expert network in the expert mixture model; based on the feature matrix and the expert weight matrix, calculate the similarity between the computing task to be processed and each expert network in the expert mixture model in parallel in the accelerator chip.

[0109] In some embodiments of the present application, based on the aforementioned scheme, the acquisition unit 301 is configured to: calculate in parallel in the accelerator chip the similarity between the computing task to be processed and each expert network in the expert mixture model according to a Hamming distance algorithm, a cosine similarity algorithm, or a gated network.

[0110] In some embodiments of the present application, based on the aforementioned solution, the expert weight matrices of the respective expert networks in the mixture of experts model are pre-stored in a distributed lookup table of the accelerator chip.

[0111] In some embodiments of the present application, based on the aforementioned solution, the apparatus further includes: an updating unit, configured to update the expert weight matrix stored in the distributed lookup table in real time based on a boundary scan test interface.

[0112] In some embodiments of the present application, based on the aforementioned solution, the apparatus further comprises: a storage unit configured to, in response to a network parameter update instruction in the on-chip memory, count the frequency of each expert network in the mixture of experts model performing computational tasks historically; define a set number of expert networks with the highest frequency as the first expert network, and define the expert networks in the mixture of experts model other than the first expert network as the second expert network; store the network parameters of the first expert network in the on-chip memory of the accelerator chip, and store the network parameters of the second expert network in the off-chip memory of the accelerator chip.

[0113] In some embodiments of the present application, based on the aforementioned scheme, the device further includes: a triggering unit, configured to trigger an update instruction of the network parameters in the on-chip memory before calling the first network parameters of the first expert network in the target expert network from the on-chip memory and calling the second network parameters of the second expert network in the target expert network from the off-chip memory.

[0114] In some embodiments of the present application, based on the aforementioned solution, the device further includes: an abstraction unit, configured to abstract the common operators in the accelerator chip into configurable modules; and based on a time division multiplexing algorithm, running each target expert network through the configurable modules.

[0115] In some embodiments of the present application, based on the aforementioned solution, the accelerator chip includes any one of a field programmable gate array chip, a graphics processing chip, a Google tensor processing chip, and a dedicated integrated circuit.

[0116] Based on the same inventive concept, an embodiment of the present application provides a computer program product, which includes computer instructions, which are stored in a computer-readable storage medium and are suitable for being read and executed by a processor, so that a computer device with the processor executes the operations performed to implement the above-mentioned acceleration method for performing computing tasks of the expert mixture model.

[0117] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, which stores at least one computer program instruction, and the at least one computer program instruction is loaded and executed by a processor to implement the operations performed by the above-mentioned acceleration method for the expert mixture model to perform computing tasks.

[0118] Based on the same inventive concept, the present application also provides an electronic device, referring to Figure 4, shows a schematic diagram of the structure of an electronic device in an embodiment of the present application, wherein the electronic device includes one or more memories 404, one or more processors 402, and at least one computer program (computer program instruction) stored in the memory 404 and executable on the processor 402. When the processor 402 executes the computer program, the acceleration method for executing computing tasks by the expert mixture model as described above is implemented.

[0119] Among them, Figure 4 In the embodiment of the present invention, a bus architecture (represented by bus 400) is shown. Bus 400 may include any number of interconnected buses and bridges. Bus 400 links various circuits together, including one or more processors represented by processor 402 and memory represented by memory 404. Bus 400 may also link various other circuits together, such as peripheral devices, voltage regulators, and power management circuits, all of which are well known in the art and, therefore, will not be described further herein. Bus interface 405 provides an interface between bus 400 and receiver 401 and transmitter 403. Receiver 401 and transmitter 403 may be the same component, namely a transceiver, which provides a unit for communicating with various other devices over a transmission medium. Processor 402 is responsible for managing bus 400 and general processing, while memory 404 may be used to store data used by processor 402 when performing operations.

[0120] The functions described herein may be implemented in hardware, software executed by a processor, firmware, or any combination thereof. If implemented in software executed by a processor, the functions may be stored as one or more instructions or codes on or transmitted via a computer-readable medium. Other examples and implementations are within the scope and spirit of this application and the appended claims. For example, due to the nature of software, the functions described above may be implemented using software executed by a processor, hardware, firmware, hardwiring, or a combination of any of these. Furthermore, the functional units may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit.

[0121] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of the units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0122] The units described as separate components may or may not be physically separate, and the components of the control device may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0123] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk, etc. Various media that can store computer program instructions.

[0124] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present application shall be included within the scope of the claims of the present application.

Claims

1. A method for accelerating the execution of computational tasks using a mixture of experts model, characterized in that: The expert mixture model is deployed in an accelerator chip, and the method includes: Obtaining a computing task to be processed, and determining a target expert network in the expert mixture model for executing the computing task to be processed, wherein network parameters of each first expert network in the expert mixture model are stored in an on-chip memory of the accelerator chip, and network parameters of each second expert network in the expert mixture model are stored in an off-chip memory of the accelerator chip; Recalling first network parameters of a first expert network in the target expert network from the on-chip memory, and recalling second network parameters of a second expert network in the target expert network from the off-chip memory; Based on the first network parameters and the second network parameters, executing the to-be-processed computing task through the target expert network in the expert mixture model; The method further includes: before calling the first network parameters of the first expert network in the target expert network from the on-chip memory and calling the second network parameters of the second expert network in the target expert network from the off-chip memory, triggering an update instruction for the network parameters in the on-chip memory; in response to the update instruction for the network parameters in the on-chip memory, counting the frequency of historical execution of computing tasks by each expert network in the expert mixture model; defining a set number of expert networks with the highest frequency as the first expert network, and defining the expert networks other than the first expert network in the expert mixture model as the second expert network; saving the network parameters of the first expert network in the on-chip memory of the accelerator chip, and saving the network parameters of the second expert network in the off-chip memory of the accelerator chip; The method of determining the target expert network in the expert mixture model for executing the computational task to be processed includes: obtaining a characteristic matrix of the computational task to be processed and an expert weight matrix of each expert network in the expert mixture model, wherein the expert weight matrix of each expert network in the expert mixture model is pre-stored in a distributed lookup table of the accelerator chip, and the expert weight matrix stored in the distributed lookup table is updated in real time based on a boundary scan test interface; based on the characteristic matrix and the expert weight matrix, the similarity between the computational task to be processed and each expert network in the expert mixture model is calculated in parallel in the accelerator chip; a set number of expert networks with the highest similarity are selected as target expert networks for executing the computational task to be processed; wherein, updating the expert weight matrix in the distributed lookup table includes: an external master controller establishes communication with the accelerator chip through a boundary scan test interface; and the new expert weight matrix is ​​written one by one or in batches into the LUT specified by the accelerator chip through a JTAG instruction, wherein the LUT content is refreshed in real time and the expert weight matrix takes effect immediately.

2. The method according to claim 1, characterized in that The parallel calculation of the similarity between the to-be-processed computing task and each expert network in the expert mixture model in the accelerator chip includes: The similarity between the to-be-processed computing task and each expert network in the expert mixture model is calculated in parallel in the accelerator chip according to the Hamming distance algorithm, the cosine similarity algorithm, or the gated network.

3. The method according to claim 1, characterized in that The method further comprises: Abstracting common operators in the accelerator chip into configurable modules; Based on a time division multiplexing algorithm, each target expert network is run through the configurable module.

4. The method according to claim 1, wherein The accelerator chip includes any one of a field programmable gate array chip, a graphics processing chip, a Google tensor processing chip, and a dedicated integrated circuit.

5. An acceleration device for performing computational tasks using a mixture of experts model, characterized in that: The expert mixture model is deployed in an accelerator chip, and the device includes: an acquisition unit, configured to acquire a computing task to be processed and determine a target expert network in the expert mixture model for executing the computing task to be processed, wherein network parameters of each first expert network in the expert mixture model are stored in an on-chip memory of the accelerator chip, and network parameters of each second expert network in the expert mixture model are stored in an off-chip memory of the accelerator chip; a calling unit, configured to call first network parameters of a first expert network in the target expert network from the on-chip memory, and call second network parameters of a second expert network in the target expert network from the off-chip memory; an execution unit, configured to execute the to-be-processed computing task through a target expert network in the mixture of experts model based on the first network parameters and the second network parameters; The device further includes: a storage unit, configured to trigger an update instruction for the network parameters in the on-chip memory before calling the first network parameters of the first expert network in the target expert network from the on-chip memory and calling the second network parameters of the second expert network in the target expert network from the off-chip memory; in response to the update instruction for the network parameters in the on-chip memory, counting the frequency of historical execution of computing tasks by each expert network in the expert mixture model; defining a set number of expert networks with the highest frequency as the first expert network, and defining the expert networks other than the first expert network in the expert mixture model as the second expert network; saving the network parameters of the first expert network in the on-chip memory of the accelerator chip, and saving the network parameters of the second expert network in the off-chip memory of the accelerator chip; The method of determining the target expert network in the expert mixture model for executing the computational task to be processed includes: obtaining a characteristic matrix of the computational task to be processed and an expert weight matrix of each expert network in the expert mixture model, wherein the expert weight matrix of each expert network in the expert mixture model is pre-stored in a distributed lookup table of the accelerator chip, and the expert weight matrix stored in the distributed lookup table is updated in real time based on a boundary scan test interface; based on the characteristic matrix and the expert weight matrix, the similarity between the computational task to be processed and each expert network in the expert mixture model is calculated in parallel in the accelerator chip; a set number of expert networks with the highest similarity are selected as target expert networks for executing the computational task to be processed; wherein, updating the expert weight matrix in the distributed lookup table includes: an external master controller establishes communication with the accelerator chip through a boundary scan test interface; and the new expert weight matrix is ​​written one by one or in batches into the LUT specified by the accelerator chip through a JTAG instruction, wherein the LUT content is refreshed in real time and the expert weight matrix takes effect immediately.

6. A computer program product, characterized in that The computer program product includes computer instructions stored in a computer-readable storage medium and adapted to be read and executed by a processor, so as to enable a computer device having the processor to perform the method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium stores at least one program code, and the at least one program code is loaded and executed by a processor to implement the operations performed by the method according to any one of claims 1 to 4.

8. An electronic device, characterized in that: The electronic device includes one or more processors and one or more memories, wherein at least one program code is stored in the one or more memories, and the at least one program code is loaded and executed by the one or more processors to implement the method according to any one of claims 1 to 4.

Images