Chip computing inference method and system of hybrid expert model

Abstract

The application provides a chip computing reasoning method and system of a mixed expert model, relates to the technical field of artificial intelligence computing, and comprises the following steps: acquiring input data to calculate expert activation weights, performing time sequence tracking and clustering analysis to identify an expert combination, and preloading the weights to a shared cache; selecting a target expert based on the activation weights, loading the weights; dividing data and constructing a pipeline level, and mapping to different computing units; controlling each unit to perform calculation according to the pipeline, and performing weighted summation to obtain a result. The application improves the reasoning efficiency and reduces the memory access overhead through expert activation mode prediction and pipeline parallel processing.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence computing technology, and in particular to a chip computing inference method and system using hybrid expert models. Background Technology

[0002] Mixture of Experts (MoE) is an advanced neural network architecture that achieves efficient and flexible reasoning capabilities through a structure of "gated network + multiple expert modules". The gated network is responsible for dynamically selecting the most suitable expert module to process the current input, while each expert module focuses on different types of data processing tasks. This design allows the model to maintain high performance while effectively controlling computational costs, and it has been widely used in large-scale language models and multimodal models in recent years.

[0003] With the diversification of artificial intelligence application scenarios, hybrid expert models have been applied to various chip platforms for inference deployment. Traditional model inference methods usually adopt static resource allocation strategies, which cannot fully adapt to the dynamic activation characteristics of expert modules in the MoE model, and cannot effectively handle the dependencies and data flow between expert modules, resulting in increased computational latency and limited overall inference throughput. Summary of the Invention

[0004] The embodiments of the present invention provide a chip computing inference method and system based on hybrid expert models, which can solve the problems in the prior art.

[0005] A first aspect of the present invention provides a chip computing inference method based on a hybrid expert model, comprising:

[0006] The input data to be inferred is obtained and input into a gating network. The activation weight distribution of multiple expert modules is calculated through the gating network.

[0007] The activation weight distribution of multiple expert modules is time-series tracked, the activation pattern sequence of each expert module in the continuous inference task is recorded and cluster analysis is performed to identify expert module combinations, and the weight parameters of the expert module combinations are preloaded into the shared cache area of ​​the chip.

[0008] Based on the activation weight distribution, select the target expert module to be activated, and load the corresponding weight parameters from the shared cache area or external memory into the calculation unit.

[0009] The input data to be inferred is divided into multiple data slices. A pipeline is constructed based on the computational dependency between the target expert module and the data slices. Parallel-executable data slices and the target expert module are mapped to different computing units, and intermediate result registers are set.

[0010] The target expert modules of each computing unit are controlled to perform inference calculations according to the pipeline level, and the inference results of each target expert module are obtained. The inference results are weighted and summed according to the activation weight distribution to obtain the final inference result.

[0011] The input data to be inferred is input into a gating network, and the activation weight distribution of multiple expert modules is calculated through the gating network, including:

[0012] The input data to be inferred is input into the gating network, and the input feature representation is extracted through the feature encoding layer of the gating network;

[0013] An expert capability matching evaluation matrix is ​​constructed based on the input feature representation, wherein each element of the expert capability matching evaluation matrix represents the degree of adaptation between different feature components in the input feature representation and the processing capabilities of each expert module;

[0014] The expert ability matching degree evaluation matrix is ​​subjected to sparsity constraint processing. By performing sparsity constraint processing, elements in the expert ability matching degree evaluation matrix whose fit degree is lower than the fit threshold are suppressed, and elements whose fit degree is higher than the fit threshold are retained, thus obtaining a sparse matching matrix.

[0015] Based on the sparse matching matrix, the input feature representation is reorganized to obtain directional feature representations for each expert module. The directional feature representations are then input into the weight generation layer of the gated network, and matrix multiplication is performed on the directional feature representations to obtain the activation weight distribution of multiple expert modules.

[0016] The activation weight distribution of multiple expert modules is time-series tracked, the activation pattern sequence of each expert module in continuous inference tasks is recorded and clustered, expert module combinations are identified, and the weight parameters of the expert module combinations are preloaded into the chip's shared cache area, including:

[0017] During the execution of a continuous inference task, the activation weight distribution corresponding to each inference task is obtained, the expert module identifiers whose weight values ​​exceed the activation judgment threshold are extracted from the activation weight distribution, and recorded in the temporal order of the inference task to form an activation mode sequence for each expert module.

[0018] Temporal pattern mining is performed on the activation pattern sequence, the activation pattern sequence is divided into multiple temporal segments, the identifiers of expert modules that are simultaneously activated in each temporal segment are extracted, the co-occurrence frequency is counted, and an expert module co-occurrence relationship matrix is ​​constructed.

[0019] Based on the co-occurrence relationship matrix of the expert modules, the correlation strength index between each pair of expert modules is calculated. The correlation strength index is used as a clustering feature to perform cluster analysis on the multiple expert modules to obtain multiple expert module clusters. The frequency of occurrence of each expert module cluster in the activation mode sequence is counted. Expert module clusters with a frequency higher than the frequency threshold are selected as expert module combinations.

[0020] Obtain the weight parameters corresponding to each expert module in the expert module combination, load the weight parameters from the external memory into the chip's shared cache area, and establish a mapping relationship between the identifier of the expert module combination and the storage address in the shared cache area.

[0021] The input data to be inferred is divided into multiple data slices. A pipeline is constructed based on the computational dependencies between the target expert module and the data slices. Parallel-executable data slices and the target expert module are mapped to different computational units, and intermediate result registers are set, including:

[0022] Based on the data distribution characteristics of the input data to be reasoned, the input data to be reasoned is divided into multiple data slices. The calculation operation sequence of the target expert module is extracted, and read-after-write dependency, write-after-read dependency, and write-after-write dependency between the calculation operations and the multiple data slices are constructed to generate a data dependency constraint matrix.

[0023] The data dependency constraint matrix is ​​decomposed into strongly connected components, and computational operations with cyclic dependencies and data fragments are assigned to the same pipeline-level atomic unit. The pipeline-level atomic units are topologically sorted to construct a pipeline-level sequence, and an execution time window is allocated to each pipeline level.

[0024] Identify pipeline stages with overlapping execution time windows in the pipeline stage sequence, extract data slices and target expert modules that can be executed in parallel in the pipeline stage, and map the data slices and target expert modules to different computing units;

[0025] Based on the data transfer mode between adjacent pipeline stages in the pipeline stage sequence, intermediate result registers are configured with corresponding access strategies for different data transfer modes. The access strategies include one-way write-read strategy, two-way exchange strategy, and broadcast strategy. The access strategy of the intermediate result register controls the data forwarding between pipeline stages.

[0026] Extract the computational operation sequence of the target expert module, construct read-after-write dependencies, write-after-read dependencies, and write-after-write dependencies between the computational operations and the multiple data shards, and generate a data dependency constraint matrix, including:

[0027] Obtain the computation graph representation of the target expert module, extract computation operations in topological order to construct a sequence of computation operations, perform data flow analysis on the sequence of computation operations, and construct a definition usage chain for each data segment. The definition usage chain records the set of computation operations that perform definition operations on the data segment and the set of computation operations that perform usage operations.

[0028] Traverse the definition and usage chain of each data shard. For each data shard, calculate the reachable definition set and the corresponding usage operation, and establish a read-after-write dependency relationship including the definition operation, data shard, and usage operation. Calculate the active usage set and subsequent definition operation of each data shard, and establish a write-after-read dependency relationship including the usage operation, data shard, and definition operation. Extract multiple definition operations of each data shard, and establish a write-after-write dependency relationship of the data shard between the preceding definition operation and the subsequent definition operation.

[0029] Traverse the read-after-write dependency, the write-after-read dependency, and the write-after-write dependency to construct a data dependency constraint matrix. The row index of the data dependency constraint matrix corresponds to the calculation operation, the column index corresponds to the data sharding, and the matrix elements represent the numerical encoding of different dependencies.

[0030] The target expert modules of each computing unit are controlled to perform inference calculations according to the pipeline level, and the inference results of each target expert module are obtained, including:

[0031] The pipeline level generates a data flow graph for inference computation. The data flow graph includes data nodes and computation nodes. Data nodes represent data shards, computation nodes represent computation operations in the pipeline level, and directed edges represent the propagation path of data shards from previous computation nodes to subsequent computation nodes.

[0032] Based on the data flow graph, an inference instruction sequence is generated for each computing unit, and each inference instruction carries the storage address of the corresponding data slice in the intermediate result register.

[0033] The inference instruction sequence is distributed to each computing unit. Each computing unit performs a data reading operation according to the inference instruction sequence, performs forward propagation calculation with the calculation operation of the corresponding target expert module, and writes the intermediate output data generated by the calculation into the intermediate result register.

[0034] During the inference computation execution process, a version identifier sequence of data fragments is maintained. When the computing unit reads data, the version identifier sequence is verified to ensure that the read data fragment is located at the correct pipeline level.

[0035] Once all nodes in the data flow graph have been calculated, the intermediate output data of each target expert module is extracted from the intermediate result register based on the output nodes identified by each pipeline level in the data flow graph, thereby obtaining the inference results of each target expert module.

[0036] During inference computation execution, a version identifier sequence for data fragments is maintained. When a computation unit reads data, the version identifier sequence is verified to ensure that the read data fragment is located at the correct pipeline level, including:

[0037] A version vector clock is maintained for each computing unit. The number of its dimensions is equal to the total number of computing units. Each dimension component records the pipeline-level operation count executed by the corresponding computing unit. When a computing unit performs a data sharding write operation, the value of its corresponding component in the version vector clock is incremented.

[0038] When a computing unit performs a write operation on a data slice, it appends a copy of the current version vector clock to the version identifier of the data slice to form a timestamp record of the version identifier sequence. The timestamp record represents the execution status of each computing unit when the data slice is written.

[0039] When a computing unit is ready to read a data shard, it obtains its own version vector clock and the timestamp record in the latest version identifier of the data shard, and performs a causal relationship determination on the two version vectors. If all component values ​​in the timestamp record are not greater than the corresponding component values ​​of its own version vector clock, it is determined that the causal consistency condition is met, and the read operation is allowed; if there is a component value in the timestamp record that is greater than the corresponding component value of its own version vector clock, it is determined that there is an unobserved concurrent write operation, and the read operation is suspended and waits for the execution progress of the computing unit corresponding to the component value to catch up.

[0040] During the waiting process, subscribe to the version vector clock update events of the relevant computing units. When a version vector clock update notification is received, re-execute the causal relationship determination until the causal consistency condition is met.

[0041] A second aspect of the present invention provides a chip computing inference system with a hybrid expert model, comprising:

[0042] The first unit is used to acquire the input data to be reasoned and input the input data to be reasoned into the gating network, and calculate the activation weight distribution of multiple expert modules through the gating network;

[0043] The second unit is used to perform time-series tracking of the activation weight distribution of multiple expert modules, record the activation pattern sequence of each expert module in the continuous inference task and perform cluster analysis, identify expert module combinations, and preload the weight parameters of the expert module combinations into the chip's shared cache area.

[0044] The third unit is used to select the target expert module to be activated based on the activation weight distribution, and load the corresponding weight parameters from the shared cache area or external memory to the calculation unit.

[0045] The fourth unit is used to divide the input data to be reasoned into multiple data slices, construct a pipeline based on the computational dependency between the target expert module and the data slices, map the parallel executable data slices and the target expert module to different computing units, and set intermediate result registers.

[0046] The fifth unit controls the target expert modules of each computing unit to perform inference calculations according to the pipeline level, obtains the inference results of each target expert module, and performs a weighted summation of each inference result according to the activation weight distribution to obtain the final inference result.

[0047] A third aspect of the embodiments of the present invention,

[0048] An electronic device is provided, comprising:

[0049] processor;

[0050] Memory used to store processor-executable instructions;

[0051] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0052] Fourth aspect of the present invention,

[0053] A computer-readable storage medium is provided, having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0054] The beneficial effects of this application are as follows:

[0055] By performing time-series tracking and cluster analysis on the activation weight distribution of expert modules, combinations of expert modules that are frequently activated together are identified, and their weight parameters are preloaded into the chip's shared cache area, which significantly reduces parameter loading latency during model inference and improves cache hit rate.

[0056] The dynamic expert module selection mechanism based on activation weight distribution enables on-demand loading of model parameters, avoiding memory waste caused by loading all expert module parameters and reducing memory bandwidth pressure.

[0057] The input data is divided into multiple data shards, and a pipeline is built based on the computational dependencies between expert modules and data shards. This enables the parallel execution of multiple expert modules on different computing units, significantly improving computational throughput.

[0058] This method dynamically adjusts the allocation of computing resources according to the characteristics of the actual reasoning task, avoiding the problem of a large number of expert modules being idle in traditional hybrid expert models, and improving the overall utilization efficiency of chip computing resources. Attached Figure Description

[0059] Figure 1 This is a flowchart illustrating the chip computing inference method using a hybrid expert model according to an embodiment of the present invention.

[0060] Figure 2 This is a flowchart illustrating the pipeline-level method for constructing computational dependencies according to an embodiment of the present invention. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0062] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0063] Figure 1 This is a flowchart illustrating the chip computing inference method using a hybrid expert model according to an embodiment of the present invention, as shown below. Figure 1 As shown, the method includes:

[0064] The input data to be inferred is obtained and input into a gating network. The activation weight distribution of multiple expert modules is calculated through the gating network.

[0065] The activation weight distribution of multiple expert modules is time-series tracked, the activation pattern sequence of each expert module in the continuous inference task is recorded and cluster analysis is performed to identify expert module combinations, and the weight parameters of the expert module combinations are preloaded into the shared cache area of ​​the chip.

[0066] Based on the activation weight distribution, select the target expert module to be activated, and load the corresponding weight parameters from the shared cache area or external memory into the calculation unit.

[0067] The input data to be inferred is divided into multiple data slices. A pipeline is constructed based on the computational dependency between the target expert module and the data slices. Parallel-executable data slices and the target expert module are mapped to different computing units, and intermediate result registers are set.

[0068] The target expert modules of each computing unit are controlled to perform inference calculations according to the pipeline level, and the inference results of each target expert module are obtained. The inference results are weighted and summed according to the activation weight distribution to obtain the final inference result.

[0069] In one optional implementation, the input data to be inferred is input into a gating network, and the activation weight distribution of multiple expert modules is calculated through the gating network, including:

[0070] The input data to be inferred is input into the gating network, and the input feature representation is extracted through the feature encoding layer of the gating network;

[0071] An expert capability matching evaluation matrix is ​​constructed based on the input feature representation, wherein each element of the expert capability matching evaluation matrix represents the degree of adaptation between different feature components in the input feature representation and the processing capabilities of each expert module;

[0072] The expert ability matching degree evaluation matrix is ​​subjected to sparsity constraint processing. By performing sparsity constraint processing, elements in the expert ability matching degree evaluation matrix whose fit degree is lower than the fit threshold are suppressed, and elements whose fit degree is higher than the fit threshold are retained, thus obtaining a sparse matching matrix.

[0073] Based on the sparse matching matrix, the input feature representation is reorganized to obtain directional feature representations for each expert module. The directional feature representations are then input into the weight generation layer of the gated network, and matrix multiplication is performed on the directional feature representations to obtain the activation weight distribution of multiple expert modules.

[0074] In this specific embodiment, the gating network in the hybrid expert reasoning system acts as a scheduler, responsible for dynamically allocating the participation ratio of different expert modules. The input data to be reasoned is input into the gating network, and the input feature representation is extracted through the feature encoding layer. Specifically, the feature encoding layer can adopt a multilayer perceptron structure, which includes a combination of several fully connected layers and nonlinear activation functions. Taking a text processing task as an example, the input data can be a sequence of word embeddings of text. The feature encoding layer will transform and aggregate these word embeddings to extract feature vectors containing semantic information. Assuming that the input feature dimension is din, the input feature representation obtained after processing by the feature encoding layer has a dimension of dh. This feature representation can capture the key features of the input data.

[0075] Based on the extracted input feature representation, an expert capability matching evaluation matrix is ​​constructed. This matrix is ​​designed to measure the degree of fit between different components of the input features and the processing capabilities of each expert module. Specifically, the input feature representation is mapped to the expert capability space through a transformation matrix W, where W has dimensions dh×(n×de), where n represents the number of expert modules and de represents the dimension of each expert's capability representation. The transformed result can be reshaped into a matching evaluation matrix M with dimensions dh×n×de. Each element M in matrix M... ijk This represents the degree of matching between the i-th input feature component and the j-th expert on the k-th capability dimension.

[0076] To improve system efficiency and expert utilization, the expert capability matching evaluation matrix is ​​subjected to sparsification constraints. The sparsification process employs a threshold filtering mechanism, setting a matching threshold τ. When an element value in the matrix is ​​less than τ, it is set to zero, while elements with values ​​greater than or equal to τ are retained. This process can be represented as applying a masking operation to matrix M to obtain a sparse matching matrix S, where S... ijk = M ijk × I(M ijk ≥ τ), where I is the indicator function. The adaptation threshold τ can be a preset fixed value or dynamically determined through an adaptive mechanism, such as taking the first k% of the largest values ​​of the elements in matrix M as the threshold. The sparsity constraint processing ensures that each input feature component is connected only with the few most matched experts, avoiding the waste of computing resources.

[0077] Based on the sparse matching matrix, the input feature representation is reorganized to obtain directional feature representations for each expert module. The feature reorganization process requires normalizing the sparse matching matrix S to ensure that the sum of the weights of each feature component is 1. The normalized sparse matrix is ​​then weighted and combined with the input feature representation to generate a directional feature representation D for each expert. Specifically, for the j-th expert module, its directional feature representation D... j It is a weighted combination of input feature representations, with weights derived from the columns in the sparse matching matrix corresponding to the expert. Each expert module receives input features that are more closely matched to its processing capabilities.

[0078] The directional feature representation is input into the weight generation layer of the gated network. Matrix multiplication is used to obtain the activation weight distribution of multiple expert modules. The weight generation layer contains a fully connected layer with a parameter matrix V of dimension de×1. For each expert j, its directional feature representation D is... j Multiplying by V yields the scalar value a. jLet represent the initial activation strength of the expert. The activation strengths of all experts are normalized using the softmax function to obtain the final activation weight distribution w = softmax([a1, a2, ..., a...). n ]), activate each element w in the weight distribution w j This represents the participation weight of the j-th expert module in the current inference task.

[0079] In practical applications, the above process can be adjusted according to the characteristics of the task. For example, in scenarios with limited computing resources, the strength of sparsity constraints can be increased to further reduce the number of activated experts; in scenarios with high precision requirements, sparsity constraints can be appropriately reduced to allow more experts to participate in the calculation. Through the above detailed gating mechanism, the hybrid expert system can dynamically adjust the activation weights of expert modules according to the characteristics of the input data, thereby achieving efficient allocation of computing resources and specialized processing, and improving the overall inference performance.

[0080] In one optional implementation, the activation weight distribution of multiple expert modules is time-series tracked, the activation pattern sequence of each expert module in a continuous inference task is recorded and clustered, expert module combinations are identified, and the weight parameters of the expert module combinations are preloaded into the chip's shared cache area, including:

[0081] During the execution of a continuous inference task, the activation weight distribution corresponding to each inference task is obtained, the expert module identifiers whose weight values ​​exceed the activation judgment threshold are extracted from the activation weight distribution, and recorded in the temporal order of the inference task to form an activation mode sequence for each expert module.

[0082] Temporal pattern mining is performed on the activation pattern sequence, the activation pattern sequence is divided into multiple temporal segments, the identifiers of expert modules that are simultaneously activated in each temporal segment are extracted, the co-occurrence frequency is counted, and an expert module co-occurrence relationship matrix is ​​constructed.

[0083] Based on the co-occurrence relationship matrix of the expert modules, the correlation strength index between each pair of expert modules is calculated. The correlation strength index is used as a clustering feature to perform cluster analysis on the multiple expert modules to obtain multiple expert module clusters. The frequency of occurrence of each expert module cluster in the activation mode sequence is counted. Expert module clusters with a frequency higher than the frequency threshold are selected as expert module combinations.

[0084] Obtain the weight parameters corresponding to each expert module in the expert module combination, load the weight parameters from the external memory into the chip's shared cache area, and establish a mapping relationship between the identifier of the expert module combination and the storage address in the shared cache area.

[0085] In this specific embodiment, it is necessary to perform temporal tracking of the activation weight distribution of multiple expert modules during the execution of continuous inference tasks to obtain the activation weight distribution corresponding to each inference task. This activation weight distribution can be obtained by recording the weight values ​​assigned to each expert module by the gating network output during the inference process of the hybrid expert model. For a hybrid expert model containing N expert modules, the activation weight distribution generated by each inference task can be represented as a vector [w1, w2, ..., w...]. N ], where w i This represents the activation weight value of the i-th expert module.

[0086] Extract the identifiers of expert modules whose activation weights exceed the activation threshold from the activation weight distribution. Typically, an activation threshold α (e.g., α=0.2) is set. When the activation weight w of an expert module exceeds the activation threshold... i When ≥α, the expert module is considered to be effectively activated. For each inference task t, a set of activated expert modules At={i|w i ≥α}, where i is the expert module identifier. These activated expert module sets are recorded in the temporal order of the reasoning tasks to form an activation pattern sequence S={A1, A2,..., Am}, where m is the total number of consecutively executed reasoning tasks.

[0087] When performing temporal pattern mining, the activation pattern sequence S is divided into multiple temporal segments. The division method can be a fixed window size method, where every k consecutive sets of activated expert modules constitute a temporal segment, resulting in multiple temporal segments; or a sliding window method, where temporal segments are obtained by sliding with a window size k and a step size s, and the sets of expert modules that are activated at the same time are extracted in each window. For example, if the window size k=10 and the step size s=5, then the first window contains the first 10 elements in the sequence, the second window contains the 6th to 15th elements, and so on.

[0088] For each time segment p, the co-occurrence of expert modules within the segment is statistically analyzed. The co-occurrence count (i,j) of two expert modules i and j is defined as the sum of the number of times they appear together in the same set of activated expert modules At. Based on the statistical results, an expert module co-occurrence relation matrix C is constructed, where the matrix element C[i][j] represents the co-occurrence count of expert modules i and j.

[0089] Based on the co-occurrence relation matrix C of expert modules, the association strength index between each pair of expert modules is calculated. The association strength can be obtained by the normalized co-occurrence frequency, i.e., R[i][j]=C[i][j] / max(C), where max(C) is the maximum value in matrix C. A more complex association strength calculation method can also be used, such as considering the activation frequency of the expert module itself, R[i][j]=C[i][j] / sqrt(C[i][i]*C[j][j]).

[0090] Using the association strength index R[i][j] as the clustering feature, cluster analysis is performed on multiple expert modules. Algorithms such as hierarchical clustering or K-means clustering can be selected. For example, when using hierarchical clustering, each expert module is initially treated as a separate cluster, and the two clusters with the highest association strength are iteratively merged until a preset clustering termination condition is met. The clustering result yields multiple expert module clusters, where each expert module cluster contains a group of highly correlated expert modules.

[0091] The frequency of each expert module cluster in the activation mode sequence is statistically analyzed. The occurrence of each expert module cluster in the set of activated expert modules At is defined as follows: if all expert modules in each expert module cluster are activated in At, it is counted as a complete occurrence; if most expert modules in each expert module cluster (e.g., more than 80%) are activated in At, it is counted as a partial occurrence. Based on the statistical results, expert module clusters with an occurrence frequency higher than the frequency threshold β are selected as expert module combinations. The frequency threshold β can be dynamically set according to system resources and performance requirements, for example, it can be taken as 10% of the total number of inference tasks.

[0092] Obtain the weight parameters corresponding to each expert module in the expert module combination. In the hybrid expert model, each expert module contains its own set of neural network weight parameters θ. i For the identified expert module combination, the set of weight parameters for all expert modules is collected.

[0093] These weight parameters are loaded from external memory (such as DRAM) into the chip's shared cache area (such as SRAM or L2 cache). Since shared cache resources are limited, they need to be optimized. The weight parameters can be trimmed or quantized according to the size and importance of the expert modules to reduce memory usage. Contiguous storage space is allocated for each expert module combination in the shared cache to facilitate fast access.

[0094] Establish a mapping between the identifiers of expert module combinations and their storage addresses in the shared cache region. This mapping can be implemented using a hash table structure, where the key is the unique identifier of the expert module combination (such as an ordered list of expert module identifiers within the combination), and the value is the starting address and size in the shared cache. During subsequent inference, when a match is detected between the currently active set of expert modules and the pre-loaded expert module combination, the weight parameters can be directly obtained from the shared cache, avoiding loading from external memory, thereby reducing memory access latency and improving inference performance.

[0095] In practical applications, the expert module combination and mapping relationships can be dynamically updated. When the workload pattern changes, time-series tracing and cluster analysis are re-executed, the expert module combination is updated, and the content and mapping relationships of the shared cache are adjusted accordingly to adapt to the new workload characteristics and maintain the system's high performance.

[0096] Figure 2 This is a flowchart illustrating the pipeline-level method for constructing computational dependencies according to an embodiment of the present invention. In one optional implementation, the input data to be inferred is divided into multiple data slices, a pipeline-level method is constructed based on the computational dependencies between the target expert module and the data slices, parallelizable data slices and the target expert module are mapped to different computational units, and intermediate result registers are set, including:

[0097] Based on the data distribution characteristics of the input data to be reasoned, the input data to be reasoned is divided into multiple data slices. The calculation operation sequence of the target expert module is extracted, and read-after-write dependency, write-after-read dependency, and write-after-write dependency between the calculation operations and the multiple data slices are constructed to generate a data dependency constraint matrix.

[0098] The data dependency constraint matrix is ​​decomposed into strongly connected components, and computational operations with cyclic dependencies and data fragments are assigned to the same pipeline-level atomic unit. The pipeline-level atomic units are topologically sorted to construct a pipeline-level sequence, and an execution time window is allocated to each pipeline level.

[0099] Identify pipeline stages with overlapping execution time windows in the pipeline stage sequence, extract data slices and target expert modules that can be executed in parallel in the pipeline stage, and map the data slices and target expert modules to different computing units;

[0100] Based on the data transfer mode between adjacent pipeline stages in the pipeline stage sequence, intermediate result registers are configured with corresponding access strategies for different data transfer modes. The access strategies include one-way write-read strategy, two-way exchange strategy, and broadcast strategy. The access strategy of the intermediate result register controls the data forwarding between pipeline stages.

[0101] In this specific embodiment, the input data is divided into multiple data segments according to the distribution characteristics of the input data to be inferred. Taking a natural language processing task as an example, the input text sequence can be divided into several segments according to semantic boundaries. Each segment contains a semantically complete sentence or paragraph. Specifically, a sliding window method can be used, with the window size set to L and the overlapping area between windows set to O. The window slides from the starting position of the input sequence to form multiple data segments.

[0102] Extract the computational operation sequence of the target expert module. Assume that the expert model contains computational units such as self-attention layer and feedforward network layer. Each computational unit corresponds to a series of operations, such as matrix multiplication and activation function calculation. For each operation, analyze its dependency relationship with the data slice and construct a data dependency constraint matrix D. The element D[i,j] in matrix D represents the dependency type between operation i and data slice j, and the value can be: read-after-write dependency (RAW), write-after-read dependency (WAR), write-after-write dependency (WAW), or no dependency (NULL).

[0103] To illustrate with a specific example, suppose the sequence of computational operations is [Op1, Op2, Op3, Op4] and the data partitions are [S1, S2, S3]. If Op1 performs a write operation on S1 and Op3 performs a read operation on S1, then there is a write-after-read dependency between Op1 and Op3. By traversing all the interaction relationships between operations and data partitions, the dependency constraint matrix D can be fully constructed.

[0104] Strongly connected component decomposition is performed on the dependency constraint matrix D to identify cyclic dependencies. The Tarjan algorithm is used to divide the operations and data fragments with cyclic dependencies into the same pipeline-level atomic unit. These atomic units are then topologically sorted to form a pipeline-level sequence [Stage1, Stage2, ...].

[0105] An execution time window is assigned to each pipeline stage. The start time of the time window is determined by the preceding dependencies, and the end time is determined by the computational complexity of that stage. For example, if the execution time of Stage 1 is t1, then the time window of Stage 1 is [0, t1]. If Stage 2 depends on the output of Stage 1, then the time window of Stage 2 will not start earlier than t1. If the execution time of Stage 2 is t2, then its window is [t1, t1+t2].

[0106] Identify pipeline stages with overlapping execution time windows in the pipeline stage sequence. For pipeline stages with overlapping windows, extract the data slices and expert modules that can be executed in parallel. The condition for determining whether data slices and expert modules can be executed in parallel is that there is no direct data dependency between them, or the dependency has been decoupled through intermediate result registers.

[0107] Parallel-executable data shards and expert modules are mapped to different computing units. For example, when the hardware platform contains multiple computing cores, independent computing tasks can be assigned to different cores. The specific mapping strategy takes into account factors such as computing load balancing, data access locality, and hardware resource limitations. Optimization algorithms (such as greedy algorithms or dynamic programming) are used to calculate the optimal mapping scheme so that the overall execution time is minimized.

[0108] Based on the data transfer pattern between adjacent pipeline stages in the pipeline sequence, configure intermediate result registers with corresponding access strategies for different data transfer patterns. The data transfer patterns include:

[0109] 1. Unidirectional delivery mode: The output of a pipeline stage is used only by a specific subsequent pipeline stage. In this mode, the intermediate result register is configured with a unidirectional write-read strategy, with write permissions granted only to the producer pipeline stage and read permissions granted only to the consumer pipeline stage.

[0110] 2. Interactive Transfer Mode: Bidirectional data exchange exists between the two pipeline stages. An intermediate result register configured with a bidirectional exchange strategy allows the two pipeline stages to alternate between read and write operations, ensuring data consistency through a semaphore mechanism.

[0111] 3. One-to-many transmission mode: The output of one pipeline stage is used by multiple subsequent pipeline stages. An intermediate result register with a broadcast strategy is configured so that after the producer pipeline stage completes its write operation, the result is automatically broadcast to all consumer pipeline stages.

[0112] For each intermediate result register, set an appropriate buffer size to avoid blocking caused by speed mismatch between producers and consumers. At the same time, implement a data forwarding mechanism so that subsequent pipeline stages can obtain the results of the previous stage as early as possible, reducing pipeline pauses.

[0113] Through the above steps, a highly efficient expert model inference execution pipeline has been constructed, achieving full utilization of computing resources and improved execution efficiency. In actual deployment, further optimizations can be made based on the specific hardware platform characteristics, such as adjusting the data sharding granularity for specific accelerators and optimizing the storage location of intermediate results, to further improve system performance.

[0114] In one optional implementation, the computational operation sequence of the target expert module is extracted, and read-after-write dependencies, write-after-read dependencies, and write-after-write dependencies are constructed between the computational operations and the multiple data shards to generate a data dependency constraint matrix, including:

[0115] Obtain the computation graph representation of the target expert module, extract computation operations in topological order to construct a sequence of computation operations, perform data flow analysis on the sequence of computation operations, and construct a definition usage chain for each data segment. The definition usage chain records the set of computation operations that perform definition operations on the data segment and the set of computation operations that perform usage operations.

[0116] Traverse the definition and usage chain of each data shard. For each data shard, calculate the reachable definition set and the corresponding usage operation, and establish a read-after-write dependency relationship including the definition operation, data shard, and usage operation. Calculate the active usage set and subsequent definition operation of each data shard, and establish a write-after-read dependency relationship including the usage operation, data shard, and definition operation. Extract multiple definition operations of each data shard, and establish a write-after-write dependency relationship of the data shard between the preceding definition operation and the subsequent definition operation.

[0117] Traverse the read-after-write dependency, the write-after-read dependency, and the write-after-write dependency to construct a data dependency constraint matrix. The row index of the data dependency constraint matrix corresponds to the calculation operation, the column index corresponds to the data sharding, and the matrix elements represent the numerical encoding of different dependencies.

[0118] In this specific embodiment, it is necessary to obtain a computation graph representation of the target expert module. This computation graph represents the computational logic inside the expert module, including computational nodes and the data flow between them. By performing topological sorting on the computation graph, it is ensured that the prerequisites of each node have been processed, forming an ordered sequence of computational operations. For example, if the target expert module includes operations such as matrix multiplication, activation function, and normalization, the operation sequence obtained after topological sorting is: input processing, weight loading, matrix multiplication, bias addition, activation function, normalization, and output processing.

[0119] Perform data flow analysis on the obtained computational operation sequence to track the definition and usage of each data slice during the computation process. Definition operations refer to operations that create or modify data slices, while usage operations are operations that read data slices without modifying their values. Construct a definition-use chain for each data slice, which can be represented in the form of: data slice identifier, set of definition operations, and set of usage operations. For example, for data slice X, its definition-use chain is: {X, {Op1, Op3},{Op2, Op4, Op5}}, indicating that X is defined by Op1 and Op3 and used by Op2, Op4, and Op5.

[0120] Traverse the definition and usage chain of each data shard to establish three types of dependencies. For read-after-write dependencies, find the reachable definition set and corresponding usage operations for each data shard. The reachable definition set refers to all defined operations that can reach a certain usage point on the execution path. If operation OpB uses data shard D, and operation OpA defines D and can reach OpB on the execution path, then establish a read-after-write dependency relationship from OpA to OpB on D, denoted as (OpA, D, OpB).

[0121] For write-after-read dependencies, calculate the active usage set of each data shard and the subsequent defined operation. If operation OpA uses data shard D, and subsequent operation OpB defines D, then establish a write-after-read dependency from OpA to OpB about D, denoted as (OpA, D, OpB). This means that the execution of OpB will change the value of D read by OpA.

[0122] For write-after-write dependencies, extract multiple definition operations for each data shard. For each pair of definition operations (OpA, OpB), if OpA is executed before OpB, then establish a write-after-write dependency relationship from OpA to OpB with respect to data shard D, denoted as (OpA, D, OpB). This means that the definition of OpB will override the definition of OpA.

[0123] Iterate through all established dependencies to construct a data dependency constraint matrix M. The row indices of the matrix correspond to operations in the computational operation sequence, the column indices correspond to data shards, and the matrix element M[i, j] represents the type of dependency between operation i and data shard j. Different numerical codes can be used to represent different types of dependencies. For example, a value of 1 represents a read-after-write dependency, a value of 2 represents a write-after-read dependency, a value of 3 represents a write-after-write dependency, and a value of 0 represents no dependency.

[0124] In practical applications, for an expert module containing three computational operations {Op1, Op2, Op3} and two data shards {D1, D2}, the dependencies are: Op1 defines D1, Op2 uses D1 and defines D2, and Op3 uses D2. Therefore, the set of read-after-write dependencies is {(Op1, D1, Op2)}, the set of write-after-read dependencies is {(Op2, D2, Op3)}, and the set of write-after-write dependencies is empty. The corresponding data dependency constraint matrix is: |D1|D2|, |----|----|----|, |Op1|1|0|, |Op2|2|1|, |Op3|0|2|, where a matrix element value of 1 indicates that the row corresponding to the operation defines the column corresponding to the data shard, a value of 2 indicates that the operation uses the data shard, and a value of 0 indicates no association.

[0125] By constructing a data dependency constraint matrix, the dependencies between computational operations and data partitions can be clearly represented, which is of great significance for subsequent parallel scheduling, memory optimization, and computational acceleration. In the parallel training of hybrid expert models, this matrix can be used to guide data exchange and synchronization operations between different expert modules, reducing unnecessary waiting and improving the utilization of computing resources.

[0126] It is worth noting that the construction of dependencies needs to consider control flow structures such as conditional branches and loops. In these cases, more complex analysis methods, such as control dependency analysis or control flow graph analysis, are required to accurately capture data dependencies. In addition, for large-scale expert modules, dependency analysis needs to use sparse matrix representation or other compression techniques to reduce memory overhead.

[0127] In one optional implementation, the target expert modules of each computing unit are controlled to perform inference calculations according to the pipeline level to obtain the inference results of each target expert module, including:

[0128] The pipeline level generates a data flow graph for inference computation. The data flow graph includes data nodes and computation nodes. Data nodes represent data shards, computation nodes represent computation operations in the pipeline level, and directed edges represent the propagation path of data shards from previous computation nodes to subsequent computation nodes.

[0129] Based on the data flow graph, an inference instruction sequence is generated for each computing unit, and each inference instruction carries the storage address of the corresponding data slice in the intermediate result register.

[0130] The inference instruction sequence is distributed to each computing unit. Each computing unit performs a data reading operation according to the inference instruction sequence, performs forward propagation calculation with the calculation operation of the corresponding target expert module, and writes the intermediate output data generated by the calculation into the intermediate result register.

[0131] During the inference computation execution process, a version identifier sequence of data fragments is maintained. When the computing unit reads data, the version identifier sequence is verified to ensure that the read data fragment is located at the correct pipeline level.

[0132] Once all nodes in the data flow graph have been calculated, the intermediate output data of each target expert module is extracted from the intermediate result register based on the output nodes identified by each pipeline level in the data flow graph, thereby obtaining the inference results of each target expert module.

[0133] In this specific embodiment, a method is provided for controlling the target expert module of each computing unit to perform inference computation. A data flow graph for inference computation is generated according to a preset pipeline level. This data flow graph is an abstract representation of the inference computation process and includes data nodes and computing nodes. Data nodes represent data slices to be processed, and computing nodes represent computational operations in the pipeline level, such as matrix multiplication and activation functions. Directed edges represent the data transmission path, indicating how data slices flow from one computing node to another. For example, for a two-layer neural network, the data flow graph may include input data nodes, first-layer computing nodes, intermediate data nodes, second-layer computing nodes, and output data nodes, which are connected by directed edges to form a complete computation path.

[0134] Based on the generated data flow graph, an inference instruction sequence is generated for each computation unit. Each inference instruction carries information required for computation, including the operation type (such as matrix multiplication, activation function application, etc.), the address of the input data in the intermediate result register, and the storage address of the output data. For example, for a matrix multiplication operation, the instruction contains the addresses of the two input matrices and the storage location of the output matrix. These instructions are organized into a sequence and ordered according to the dependencies in the data flow graph, ensuring that the preceding dependent operations are executed first, followed by the subsequent operations.

[0135] The generated inference instruction sequence is distributed to each computing unit. The distribution process takes load balancing into account, and instructions are allocated reasonably according to the processing capacity and current load of the computing unit. After receiving the instruction, each computing unit reads data from the intermediate result register according to the address information carried in the instruction and performs the corresponding calculation operation. The calculation operation is matrix multiplication, vector addition, activation function application, etc., depending on the structure of the target expert module. After the calculation is completed, the computing unit writes the generated intermediate output data back to the specified location of the intermediate result register for subsequent calculation.

[0136] Throughout the inference computation execution process, a version identifier sequence for each data shard is maintained. Each data shard has a version identifier, and its version identifier is updated accordingly when the data is updated. This mechanism ensures that in a parallel computing environment, the computing unit always reads the correct pipeline-level data. When a computing unit attempts to read a data shard, it first verifies whether the version identifier of the data shard meets the requirements of the current pipeline level. If it does not meet the requirements, it means that the data is not yet ready or has been updated, and the computing unit will wait or re-acquire the correct version of the data.

[0137] For example, suppose there is a data shard D. After being processed by compute unit A in pipeline stage 1, it produces version V1. In pipeline stage 2, it is processed by compute unit B, producing version V2. When compute unit C needs to use this data in pipeline stage 3, it verifies whether the current version of data shard D is V2. If it is V1 or another version, it means that the data has not yet been processed in pipeline stage 2, and compute unit C needs to wait for the correct version of the data.

[0138] Once all nodes in the data flow graph have been computed, the final output data of each target expert module is extracted from the intermediate result register based on the output nodes identified in the data flow graph to obtain the inference result. The output nodes are usually data nodes in the data flow graph that have no subsequent computation nodes, representing the final result of the computation. The system will identify these nodes and extract data from the corresponding address in the intermediate result register to assemble a complete inference result.

[0139] In practical applications, this method can be used for distributed inference of large language models. For example, for a Transformer model containing billions of parameters, it can be decomposed into multiple expert modules distributed across different computing units. The input text sequence is segmented into data fragments, which are then transferred between expert modules through the pipeline mechanism described above to generate text continuation or question answer results. This approach not only improves the efficiency of computing resource utilization but also reduces the latency of model inference, enabling large-scale AI models to be efficiently applied to real-time interactive scenarios.

[0140] Through the synergistic application of the above technical means, efficient control of the target expert modules of each computing unit is achieved, enabling them to perform inference calculations according to the preset pipeline level, while ensuring data consistency and calculation correctness, and ultimately obtaining reliable inference results.

[0141] In one optional implementation, a version identifier sequence of data fragments is maintained during inference computation execution. When a computation unit reads data, the version identifier sequence is verified to ensure that the read data fragment is located at the correct pipeline level, including:

[0142] A version vector clock is maintained for each computing unit. The number of its dimensions is equal to the total number of computing units. Each dimension component records the pipeline-level operation count executed by the corresponding computing unit. When a computing unit performs a data sharding write operation, the value of its corresponding component in the version vector clock is incremented.

[0143] When a computing unit performs a write operation on a data slice, it appends a copy of the current version vector clock to the version identifier of the data slice to form a timestamp record of the version identifier sequence. The timestamp record represents the execution status of each computing unit when the data slice is written.

[0144] When a computing unit is ready to read a data shard, it obtains its own version vector clock and the timestamp record in the latest version identifier of the data shard, and performs a causal relationship determination on the two version vectors. If all component values ​​in the timestamp record are not greater than the corresponding component values ​​of its own version vector clock, it is determined that the causal consistency condition is met, and the read operation is allowed; if there is a component value in the timestamp record that is greater than the corresponding component value of its own version vector clock, it is determined that there is an unobserved concurrent write operation, and the read operation is suspended and waits for the execution progress of the computing unit corresponding to the component value to catch up.

[0145] During the waiting process, subscribe to the version vector clock update events of the relevant computing units. When a version vector clock update notification is received, re-execute the causal relationship determination until the causal consistency condition is met.

[0146] In this specific embodiment, a unique identifier needs to be assigned to each computing unit and the version vector clock needs to be initialized. Assuming there are n computing units, each computing unit maintains an n-dimensional version vector clock with an initial value of zero vector. Each dimension corresponds to a specific computing unit and records the pipeline-level operation count executed by that computing unit.

[0147] When a computing unit performs a write operation, it increments the value of its own component in its version vector clock. For example, if the version vector clock of computing unit C1 is [3,1,2] (assuming there are three computing units in the system), after performing a write operation, its version vector clock is updated to [4,1,2]. This increment operation indicates that computing unit C1 has completed the fourth pipelined operation.

[0148] After performing a write operation on a data slice, the computing unit appends a copy of the current version vector clock to the version identifier of the data slice. The version identifier of the data slice is actually a sequence of version identifiers. Each write operation adds a timestamp record to the sequence. For example, the initial version identifier sequence of data slice D is empty. When computing unit C1 writes, its version identifier sequence becomes [[4,1,2]], indicating that when the data slice was last written, the execution states of computing units C1, C2, and C3 were 4, 1, and 2 pipelined operations, respectively.

[0149] When a computing unit is ready to read a data shard, it needs to perform a causal consistency check. It obtains the timestamp records in its own version vector clock and the latest version identifier of the target data shard. For example, if the version vector clock of computing unit C2 is [3,5,2] and it is ready to read data shard D (whose latest version identifier is [4,1,2]), when performing causal relationship determination, it compares the two version vectors: [3,5,2] and [4,1,2]. It finds that the first component 4 of [4,1,2] is greater than the first component 3 of [3,5,2], indicating that computing unit C2 has not yet observed the result of the fourth operation of computing unit C1, and there is a potential causal relationship violation.

[0150] Upon detecting a causal violation, the read operation is suspended. Computation unit C2 subscribes to the version vector clock update event of computation unit C1. When the execution progress of C1 is observed by other computation units (e.g., through message passing or shared memory updates), C2 will receive a notification and update its own version vector clock. Assuming that after receiving the update, C2's version vector clock becomes [4,5,2], a causal relationship determination is performed again. It is found that all components of [4,5,2] are not less than the corresponding components of [4,1,2], satisfying the causal consistency condition. Therefore, the read operation is allowed to continue.

[0151] In practical applications, an event-driven architecture can be used to implement the update notification mechanism for the version vector clock. After each computing unit updates its own version vector clock, it publishes an update event to the event bus. Other computing units subscribe to these events and update their understanding of the execution status of other computing units based on the received information. For example, when computing unit C1 updates its version vector clock from [3,1,2] to [4,1,2], it publishes the event "C1:4", indicating that C1 has completed the 4th pipelined operation. After receiving this event, other computing units update the component corresponding to C1 in their own version vector clock to 4.

[0152] To optimize performance, a batch update mechanism can be adopted in practice. The computing unit does not need to publish the update event immediately after each operation, but can publish it uniformly after completing a batch of operations. Similarly, the receiver can also use batch processing to process the update event, reducing the frequency of version vector clock updates.

[0153] When handling data shard version conflicts, a multi-version concurrency control mechanism can be adopted. When multiple computing units write to the same data shard at the same time, each write operation will add a record to the version identifier sequence. The read operation can select the latest version that meets the causal consistency condition according to its own version vector clock. For example, if the version identifier sequence of data shard D is [[3,1,2],[3,2,1]] and the version vector clock of computing unit C2 is [3,2,2], then C2 can read the second version [3,2,1] because it satisfies the causal consistency condition.

[0154] When there are a large number of data shards and computation units, maintaining and transmitting version vector clocks incurs performance overhead. To address this, an incremental update mechanism can be implemented, transmitting only the changed components of the version vector clock. Furthermore, the dimensionality of the version vector clock can be optimized based on the system topology, grouping computation units that do not interact directly into the same logical group, reducing the dimensionality of the version vector, and thus lowering storage and computational overhead. Through these mechanisms, version consistency of data shards can be effectively maintained during distributed inference computation, ensuring correct data flow between pipeline stages and improving system reliability and performance.

[0155] The chip computing inference system based on a hybrid expert model according to embodiments of the present invention includes:

[0156] The first unit is used to acquire the input data to be reasoned and input the input data to be reasoned into the gating network, and calculate the activation weight distribution of multiple expert modules through the gating network;

[0157] The second unit is used to perform time-series tracking of the activation weight distribution of multiple expert modules, record the activation pattern sequence of each expert module in the continuous inference task and perform cluster analysis, identify expert module combinations, and preload the weight parameters of the expert module combinations into the chip's shared cache area.

[0158] The third unit is used to select the target expert module to be activated based on the activation weight distribution, and load the corresponding weight parameters from the shared cache area or external memory to the calculation unit.

[0159] The fourth unit is used to divide the input data to be reasoned into multiple data slices, construct a pipeline based on the computational dependency between the target expert module and the data slices, map the parallel executable data slices and the target expert module to different computing units, and set intermediate result registers.

[0160] The fifth unit controls the target expert modules of each computing unit to perform inference calculations according to the pipeline level, obtains the inference results of each target expert module, and performs a weighted summation of each inference result according to the activation weight distribution to obtain the final inference result.

[0161] A third aspect of the present invention provides an electronic device, comprising:

[0162] processor;

[0163] Memory used to store processor-executable instructions;

[0164] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0165] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0166] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.

[0167] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A chip computational inference method based on hybrid expert models, characterized in that, include: The input data to be inferred is obtained and input into a gating network. The activation weight distribution of multiple expert modules is calculated through the gating network. The activation weight distribution of multiple expert modules is time-series tracked, the activation pattern sequence of each expert module in continuous inference tasks is recorded and clustered, expert module combinations are identified, and the weight parameters of the expert module combinations are preloaded into the chip's shared cache area, including: During the execution of a continuous inference task, the activation weight distribution corresponding to each inference task is obtained, the expert module identifiers whose weight values ​​exceed the activation judgment threshold are extracted from the activation weight distribution, and recorded in the temporal order of the inference task to form an activation mode sequence for each expert module. Temporal pattern mining is performed on the activation pattern sequence, the activation pattern sequence is divided into multiple temporal segments, the identifiers of expert modules that are simultaneously activated in each temporal segment are extracted, the co-occurrence frequency is counted, and an expert module co-occurrence relationship matrix is ​​constructed. Based on the co-occurrence relationship matrix of the expert modules, the correlation strength index between each pair of expert modules is calculated. The correlation strength index is used as a clustering feature to perform cluster analysis on the multiple expert modules to obtain multiple expert module clusters. The frequency of occurrence of each expert module cluster in the activation mode sequence is counted. Expert module clusters with a frequency higher than the frequency threshold are selected as expert module combinations. Obtain the weight parameters corresponding to each expert module in the expert module combination, load the weight parameters from external memory into the chip's shared cache area, and establish a mapping relationship between the identifier of the expert module combination and the storage address in the shared cache area; Based on the activation weight distribution, select the target expert module to be activated, and load the corresponding weight parameters from the shared cache area or external memory into the calculation unit. The input data to be inferred is divided into multiple data slices. A pipeline is constructed based on the computational dependencies between the target expert module and the data slices. Parallel-executable data slices and the target expert module are mapped to different computational units, and intermediate result registers are set, including: Based on the data distribution characteristics of the input data to be reasoned, the input data to be reasoned is divided into multiple data slices. The calculation operation sequence of the target expert module is extracted, and read-after-write dependency, write-after-read dependency, and write-after-write dependency between the calculation operations and the multiple data slices are constructed to generate a data dependency constraint matrix. The data dependency constraint matrix is ​​decomposed into strongly connected components, and computational operations with cyclic dependencies and data fragments are assigned to the same pipeline-level atomic unit. The pipeline-level atomic units are topologically sorted to construct a pipeline-level sequence, and an execution time window is allocated to each pipeline level. Identify pipeline stages with overlapping execution time windows in the pipeline stage sequence, extract data slices and target expert modules that can be executed in parallel in the pipeline stage, and map the data slices and target expert modules to different computing units; Based on the data transfer mode between adjacent pipeline stages in the pipeline stage sequence, intermediate result registers with corresponding access strategies are configured for different data transfer modes. The access strategies include one-way write-read strategy, two-way exchange strategy and broadcast strategy. The access strategy of the intermediate result register controls the data forwarding between pipeline stages. The target expert modules of each computing unit are controlled to perform inference calculations according to the pipeline level, and the inference results of each target expert module are obtained. The inference results are weighted and summed according to the activation weight distribution to obtain the final inference result.

2. The method according to claim 1, characterized in that, The input data to be inferred is input into a gating network, and the activation weight distribution of multiple expert modules is calculated through the gating network, including: The input data to be inferred is input into the gating network, and the input feature representation is extracted through the feature encoding layer of the gating network; An expert capability matching evaluation matrix is ​​constructed based on the input feature representation, wherein each element of the expert capability matching evaluation matrix represents the degree of adaptation between different feature components in the input feature representation and the processing capabilities of each expert module; The expert ability matching degree evaluation matrix is ​​subjected to sparsity constraint processing. By performing sparsity constraint processing, elements in the expert ability matching degree evaluation matrix whose fit degree is lower than the fit threshold are suppressed, and elements whose fit degree is higher than the fit threshold are retained, thus obtaining a sparse matching matrix. Based on the sparse matching matrix, the input feature representation is reorganized to obtain directional feature representations for each expert module. The directional feature representations are then input into the weight generation layer of the gated network, and matrix multiplication is performed on the directional feature representations to obtain the activation weight distribution of multiple expert modules.

3. The method according to claim 1, characterized in that, Extract the computational operation sequence of the target expert module, construct read-after-write dependencies, write-after-read dependencies, and write-after-write dependencies between the computational operations and the multiple data shards, and generate a data dependency constraint matrix, including: Obtain the computation graph representation of the target expert module, extract computation operations in topological order to construct a sequence of computation operations, perform data flow analysis on the sequence of computation operations, and construct a definition usage chain for each data segment. The definition usage chain records the set of computation operations that perform definition operations on the data segment and the set of computation operations that perform usage operations. Traverse the definition and usage chain of each data shard. For each data shard, calculate the reachable definition set and the corresponding usage operation, and establish a read-after-write dependency relationship including the definition operation, data shard, and usage operation. Calculate the active usage set and subsequent definition operation of each data shard, and establish a write-after-read dependency relationship including the usage operation, data shard, and definition operation. Extract multiple definition operations of each data shard, and establish a write-after-write dependency relationship of the data shard between the preceding definition operation and the subsequent definition operation. Traverse the read-after-write dependency, the write-after-read dependency, and the write-after-write dependency to construct a data dependency constraint matrix. The row index of the data dependency constraint matrix corresponds to the calculation operation, the column index corresponds to the data sharding, and the matrix elements represent the numerical encoding of different dependencies.

4. The method according to claim 1, characterized in that, The target expert modules of each computing unit are controlled to perform inference calculations according to the pipeline level, and the inference results of each target expert module are obtained, including: The pipeline level generates a data flow graph for inference computation. The data flow graph includes data nodes and computation nodes. Data nodes represent data shards, computation nodes represent computation operations in the pipeline level, and directed edges represent the propagation path of data shards from previous computation nodes to subsequent computation nodes. Based on the data flow graph, an inference instruction sequence is generated for each computing unit, and each inference instruction carries the storage address of the corresponding data slice in the intermediate result register. The inference instruction sequence is distributed to each computing unit. Each computing unit performs a data reading operation according to the inference instruction sequence, performs forward propagation calculation with the calculation operation of the corresponding target expert module, and writes the intermediate output data generated by the calculation into the intermediate result register. During the inference computation execution process, a version identifier sequence of data fragments is maintained. When the computing unit reads data, the version identifier sequence is verified to ensure that the read data fragment is located at the correct pipeline level. Once all nodes in the data flow graph have been calculated, the intermediate output data of each target expert module is extracted from the intermediate result register based on the output nodes identified by each pipeline level in the data flow graph, thereby obtaining the inference results of each target expert module.

5. The method according to claim 4, characterized in that, During inference computation execution, a version identifier sequence for data fragments is maintained. When a computation unit reads data, the version identifier sequence is verified to ensure that the read data fragment is located at the correct pipeline level, including: A version vector clock is maintained for each computing unit. The number of its dimensions is equal to the total number of computing units. Each dimension component records the pipeline-level operation count executed by the corresponding computing unit. When a computing unit performs a data sharding write operation, the value of its corresponding component in the version vector clock is incremented. When a computing unit performs a write operation on a data slice, it appends a copy of the current version vector clock to the version identifier of the data slice to form a timestamp record of the version identifier sequence. The timestamp record represents the execution status of each computing unit when the data slice is written. When a computing unit is ready to read a data shard, it obtains its own version vector clock and the timestamp record in the latest version identifier of the data shard, and performs a causal relationship determination on the two version vectors. If all component values ​​in the timestamp record are not greater than the corresponding component values ​​of its own version vector clock, it is determined that the causal consistency condition is met, and the read operation is allowed; if there is a component value in the timestamp record that is greater than the corresponding component value of its own version vector clock, it is determined that there is an unobserved concurrent write operation, and the read operation is suspended and waits for the execution progress of the computing unit corresponding to the component value to catch up. During the waiting process, subscribe to the version vector clock update events of the relevant computing units. When a version vector clock update notification is received, re-execute the causal relationship determination until the causal consistency condition is met.

6. A chip computing inference system based on a hybrid expert model, used to implement the method as described in any one of claims 1-5, characterized in that, include: The first unit is used to acquire the input data to be reasoned and input the input data to be reasoned into the gating network, and calculate the activation weight distribution of multiple expert modules through the gating network; The second unit is used to perform time-series tracking of the activation weight distribution of multiple expert modules, record the activation pattern sequence of each expert module in the continuous inference task and perform cluster analysis, identify expert module combinations, and preload the weight parameters of the expert module combinations into the chip's shared cache area. The third unit is used to select the target expert module to be activated based on the activation weight distribution, and load the corresponding weight parameters from the shared cache area or external memory to the calculation unit. The fourth unit is used to divide the input data to be reasoned into multiple data slices, construct a pipeline based on the computational dependency between the target expert module and the data slices, map the parallel executable data slices and the target expert module to different computing units, and set intermediate result registers. The fifth unit controls the target expert modules of each computing unit to perform inference calculations according to the pipeline level, obtains the inference results of each target expert module, and performs a weighted summation of each inference result according to the activation weight distribution to obtain the final inference result.

7. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 5.

Images