Multi-dimensional parallel artificial intelligence processor for large model reasoning

Abstract

The invention discloses a multi-dimensional parallel artificial intelligence processor for large model reasoning, relates to the technical field of artificial intelligence processors, and solves the problem that fixed and rigid hardware parallel configuration cannot adapt to a dynamically changing reasoning load. In order to solve the problems of low resource utilization rate, aggravated communication conflicts and unstable overall performance caused by the fact that a current task is not required, the number and the type of required virtual modules are accurately calculated according to the batch size and the sequence length which are changed in real time, and physical calculation units are dynamically combined according to the number and the type of the required virtual modules, so that processor resources can elastically match the actual requirements of the current task; the contradiction that small tasks occupy large resources or large task resources are insufficient under fixed configuration is fundamentally overcome, so that the average utilization rate of the computing unit is improved, and invalid power consumption is reduced.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence processor technology, specifically to a multi-dimensional parallel artificial intelligence processor for large model inference. Background Technology

[0002] With the widespread application of large language models with hundreds of billions or even trillions of parameters, their inference services need to handle highly dynamic and varied loads, such as varying user request batch sizes, differentiated input sequence lengths, and the alternating attention and feedforward network layers in the model. To accelerate such computations, the industry typically uses dedicated Tensor Processing Units (TPUs) or AI processors, which contain tens of thousands of computing units. To utilize these massive units, they are organized to work collaboratively through preset, fixed parallel configuration strategies (such as fixed tensor parallel mesh partitioning and pipeline stages). However, this static and fixed hardware resource configuration architecture reveals significant shortcomings when facing dynamically changing inference loads:

[0003] When processing small batches or short sequences of requests, a large number of computing units are idle due to the mismatch between task partitioning granularity and hardware configuration, resulting in low overall computing resource utilization and poor energy efficiency. Conversely, when processing large batches or long sequences of requests, fixed communication paths and limited on-chip memory bandwidth become bottlenecks. Severe memory access conflicts and communication congestion occur between different computing units or computing groups due to resource contention, leading to a decrease in overall throughput and an increase in inference latency. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a multi-dimensional parallel artificial intelligence processor for large-scale model inference, which solves the problems of low resource utilization, increased communication conflicts, and unstable overall performance caused by the inability of fixed and rigid hardware parallel configurations to adapt to dynamically changing inference loads.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a multi-dimensional parallel artificial intelligence processor for large model inference, comprising:

[0006] The resource management terminal is used to identify all physical computing units and obtain the undirected graph G formed by the physical communication links between units.

[0007] The load decision-making end calculates the required virtual module type and the final number of virtual modules Nfn based on the batch size, sequence length, and computational layer class of the current model task.

[0008] On the module construction side, all idle units are dynamically clustered into an optimized resource pool with high communication efficiency based on the undirected graph G. Then, based on the module type and Nfn, all required units are selected from the same pool for construction first. If resources are insufficient, they are supplemented from adjacent pools.

[0009] The operator mapping end is used to parse the model computation graph. Based on the data dependency strength and communication cost between operators, it prioritizes mapping operators to the virtual module where their strongly dependent predecessor is located. If the module is overloaded, it will be mapped to other ready modules with the closest physical location.

[0010] The performance monitoring terminal periodically collects the computing utilization and communication load of each virtual module, and triggers the merging or splitting of virtual modules based on preset thresholds and durations.

[0011] As a further aspect of the present invention: the resource management terminal is specifically used for:

[0012] Each physical computing unit is divided into basic computing units and functional computing units, and each unit is assigned a unique physical identifier (PID) and type label.

[0013] Record the unit type, current status, and the identifier VID of the virtual module to which each PID belongs when it is occupied, as a global resource status table;

[0014] By sending a neighbor discovery request to all computing units, each computing unit reports the physical identifiers of its neighboring computing units, constructing an undirected graph G=(V,E). In the graph, each vertex v∈V represents a physical unit (identified by its PID), and each edge e(u,v)∈E indicates that there is a direct physical communication link between units u and v.

[0015] As a further aspect of the present invention: the computational layer type is an attention layer or a feedforward network layer;

[0016] If the current layer is an attention layer, then calculate the theoretically required number of virtual attention modules Nth = ceil(Ha / Hpm) * ceil(Bs / Bpa), where Ha is the number of attention heads, Bs is the batch size, and Hpm and Bpa are preset capability parameters;

[0017] If the current layer is a feedforward network layer, then the theoretical number of virtual modules required, Nth, is calculated as Nth = ceil(D / Dpm) * ceil(Bs / Bpf), where D is the hidden layer dimension, and Dpm and Bpf are preset capability parameters.

[0018] Get the total number of currently idle and matching physical units Nav, and calculate the maximum buildable limit Nal = floor(Nav / Upm), where Upm is the number of core units required to build the target module.

[0019] Determine the final number of builds Nbd = min(Nth, Nal), and align Nbd to a power of 2 that is no greater than its maximum power of 2 to obtain the final number of builds Nfn.

[0020] As a further aspect of the present invention: dynamically clustering the idle units into an optimized resource pool for efficient communication includes:

[0021] Each idle physical unit is used as an independent initial resource pool;

[0022] Traverse all resource pools. For any two resource pools Pi and Pj, calculate their merged communication cost C = α * (average shortest path length between all cell pairs in Pi and Pj in the undirected graph G) + β * (maximum value of all shortest path lengths), where α and β are preset weighting coefficients.

[0023] Select the pair with the smallest C value among all currently mergeable pools and merge them into a new resource pool.

[0024] When the number of remaining resource pools decreases to the preset target value K, or the current minimum C value exceeds the preset threshold, merging stops, and K optimized resource pools are finally formed.

[0025] As a further aspect of the present invention: the operator mapping layer is specifically used for:

[0026] Analyze the computational graph of the analytical model, extract operator nodes and their data dependencies, and construct a directed data dependency graph;

[0027] For the computation layer that requires the allocation of virtual modules, analyze its data dependency graph, obtain the critical path from input to output, and generate a priority mapping queue Qcri according to the execution order of the operators on the critical path;

[0028] In the remaining subgraph, repeatedly remove nodes with an in-degree of zero to generate a secondary queue;

[0029] Prioritize mapping the current operator to the virtual module containing its predecessor operator with the strongest communication strength to achieve operator fusion;

[0030] If the target module load exceeds a preset threshold, then the other ready module in the undirected graph G that is closest to the target module in terms of physical location and has the lightest load is selected for mapping.

[0031] As a further aspect of the present invention: the operator mapping layer is also used to process broadcast operators:

[0032] Identify broadcast operators whose output is input to multiple downstream operators, and obtain the set of target virtual modules to which all their direct successor operators have been mapped;

[0033] Find a virtual module Vctr with a central location in an undirected graph G such that the sum of the hop counts from Vctr to each module in the set is minimized, and map the broadcast operator to Vctr.

[0034] As a further aspect of the present invention: the performance monitoring layer is specifically used for:

[0035] The computational utilization rate U and the output communication queue depth Q of each virtual module are collected at fixed intervals.

[0036] If the average utilization of two or more virtual modules is detected to be less than the low utilization threshold, and the average queue depth of all data channels between modules is lower than the threshold Qlow, and this continues for Tlow monitoring cycles, then a module merging operation is triggered.

[0037] As a further aspect of the present invention: the module merging operation includes:

[0038] Tracing back the origin of the computation graph from the inefficient module set M, we can identify whether they originate from the same computation subgraph;

[0039] For the identified homologous module groups, the affinity score A = γ1*(1-∑Finter / Tmon) + γ2*SPR is calculated, where γ1 and γ2 are preset weights, ∑Finter is the total amount of communication in the historical period, Tmon is the monitoring period, and SPR is the proportion of shared direct predecessors.

[0040] Select the combination with the highest affinity score as the candidate set to be merged, take the module with the highest load as the main module Vm, and merge the operator subgraphs of other candidate modules into the computation flow of the main module.

[0041] Dissolve the other modules that were merged, update the status of other physical units to idle, and update the on-chip network routing table.

[0042] As a further aspect of the present invention: the performance monitoring layer is also used for:

[0043] If the computational utilization U of a single virtual module is detected to be greater than the high load threshold Uhi, and the output communication queue depth Q is greater than the congestion threshold Qth, and this continues for Thigh monitoring cycles, then a module split operation is triggered.

[0044] As a further aspect of the present invention: the module splitting operation includes:

[0045] Analyze the operator subgraph within this module to identify the critical path;

[0046] In non-critical paths, candidate branches Br with large computational load, inputs mainly from intermediate results within the module, and outputs mainly used for subsequent calculations within the module are selected as the operator set;

[0047] For each Br, obtain the revenue Ben and the cost Cos;

[0048] Select Br, which has the highest benefit-cost ratio Rbc=Ben / Cos, as the splitting target;

[0049] A new module is built from the idle resource pool allocation unit, and the selected branch Br is migrated to the new module.

[0050] This invention provides a multi-dimensional parallel artificial intelligence processor for large-scale model inference. Compared with existing technologies, it has the following advantages:

[0051] (1) Based on the real-time changing batch size and sequence length, the present invention accurately calculates the required number and type of virtual modules, and dynamically combines physical computing units accordingly, so that processor resources can be flexibly matched with the actual needs of the current task, fundamentally overcoming the contradiction of small tasks occupying large resources or large tasks having insufficient resources under fixed configuration, thereby improving the average utilization rate of computing units and reducing ineffective power consumption.

[0052] (2) By prioritizing the combination of units with close physical location and low communication latency when constructing virtual modules, this invention optimizes the internal communication path of the module. At the same time, it aggregates operators with strong data dependencies to be executed in the same module and places broadcast operations in the center of the network, thereby minimizing unnecessary on-chip and off-chip data movement, reducing memory access conflicts and communication latency caused by resource contention, and improving the overall data processing throughput.

[0053] (3) This invention automatically identifies the state of low or overloaded resource utilization and triggers merging or splitting operations to achieve elastic scaling of resources. Combined with the cooling period, it can adapt to long-term load trends while suppressing frequent oscillations caused by instantaneous fluctuations, ensuring long-term stability and high efficiency of performance under high dynamic load environments. Attached Figure Description

[0054] Figure 1 This is a flowchart illustrating the overall workflow of the present invention. Detailed Implementation

[0055] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] Example 1

[0057] Please see Figure 1This application provides a multi-dimensional parallel artificial intelligence processor for large model inference, including:

[0058] The resource management terminal is used to identify all physical computing units and obtain the undirected graph G formed by the physical communication links between units.

[0059] The load decision-making end calculates the required virtual module type and the final number of virtual modules Nfn based on the batch size, sequence length, and computational layer class of the current model task.

[0060] On the module construction side, all idle units are dynamically clustered into an optimized resource pool with high communication efficiency based on the undirected graph G. Then, based on the module type and Nfn, all required units are selected from the same pool for construction first. If resources are insufficient, they are supplemented from adjacent pools.

[0061] The operator mapping end is used to parse the model computation graph. Based on the data dependency strength and communication cost between operators, it prioritizes mapping operators to the virtual module where their strongly dependent predecessor is located. If the module is overloaded, it will be mapped to other ready modules with the closest physical location.

[0062] The performance monitoring terminal periodically collects the computing utilization and communication load of each virtual module, and triggers the merging or splitting of virtual modules based on preset thresholds and durations.

[0063] Example 2

[0064] This illustrates another embodiment of the present invention. Based on Embodiment 1, the content of Embodiment 1 is described in further detail. This application provides a multi-dimensional parallel artificial intelligence processor for large model inference, specifically including:

[0065] S1: Scan and identify all physical computing units within the processor, and classify them into basic computing units (CE) and functional computing units (FE) based on the core instruction set supported by each unit. The basic computing unit (CE) is dedicated to performing general matrix multiplication and addition operations, and the basic computing unit (FE) is used to perform special operations such as activation functions and normalization. At the same time, a unique physical identifier (PID) is assigned to each basic computing unit (CE) and functional computing unit (FE), and a type label is written into the status register inside the unit according to its type.

[0066] Meanwhile, the resource management unit records the unit type and current status (idle / occupied) corresponding to each PID. If it is occupied, it records the identifier (VID) of the virtual module to which it belongs, so as to build a global physical resource status table.

[0067] A request is sent to all computing units. After each computing unit accepts the request, it reports the physical identifiers of its neighboring computing units, thus constructing an undirected graph G=(V,E). In the graph, each vertex v∈V represents a physical unit (identified by its PID), and each edge e(u,v)∈E indicates that there is a direct physical communication link between units u and v. The weight W(e) of the edge represents the normalized communication delay or hop count.

[0068] S2: Real-time acquisition of the feature parameters of the current and upcoming tasks, including: the number of samples in the current batch (batch size B), the length of the input sequence (sequence length L), the type of computation layer to be executed: attention layer, feedforward network layer, the model parameters D of the layer, and the number of attention heads H (if it is an attention layer).

[0069] Based on the acquired real-time feature parameters, the required computation is triggered at the start of each computation layer scheduling:

[0070] If it is an attention layer, then calculate the theoretical number of virtual attention modules Nth = ceil(Ha / Hpm) * ceil(Bs / Bpa), where Hpm (e.g. 4) represents the maximum number of attention heads that a virtual attention module can process in parallel, Bpa (e.g. 2) represents its parallel processing capability for batch samples, Ha is the number of attention heads in the current attention layer, and Bs is the batch size of the current inference task, i.e. the number of input samples processed in parallel at one time.

[0071] If it is a feedforward network layer, the theoretical requirement Nth is calculated as Nth = ceil(D / Dpm) * ceil(Bs / Bpf), where D is the hidden layer dimension of the current feedforward network layer, and Dpm is the number of dimensions that each virtual module can handle (e.g., 512).

[0072] Get the total number of physical units Nav that are currently idle and of the same type. Calculate the upper limit of the number of target virtual modules that can be built under the current conditions, Nal=floor(Nav / Upm), where Upm is the number of core units required to build a single target module. Finally, determine the number of modules to be built, Nbd=min(Nth, Nal).

[0073] Alignment optimization is performed on Nbd by adjusting its value to a power of 2 that is no greater than Nbd (e.g., 1, 2, 4, 8...), resulting in the final number of builds Nfn.

[0074] The load decision layer sends a resource allocation instruction package containing the target type of the virtual module, the number of builds Nfn, and the expected completion time to the resource management layer.

[0075] S3: Treat all currently idle physical units as an independent resource pool, traverse all resource pools, and for any two resource pools Pi and Pj, calculate their merging communication cost C. The specific calculation formula is: C = α * (average shortest path length between all unit pairs in Pi and Pj in the undirected graph G) + β * (maximum value of all the above shortest path lengths), where α and β are preset weighting coefficients.

[0076] Select the pair with the smallest C value among all currently mergeable pools and merge them into a new resource pool.

[0077] When the number of remaining resource pools decreases to the preset target value K, or the current minimum C value exceeds the preset threshold, merging stops, and K optimized resource pools are finally formed. The physical units of each pool's content have a high degree of connection density.

[0078] Upon receiving the final number of builds Nfn, allocate resources for each virtual module to be built: based on the type and quantity of units required by the module, prioritize selecting all required idle units from the same optimized resource pool;

[0079] If a single pool of resources is insufficient, it will be supplemented by selecting from other resource pools that are directly adjacent to the pool in the topology (connected by an edge).

[0080] In the global physical resource status table, the PID of these units is bound to the VID, the status is updated to occupied, and their VID, member list, and capability attributes are recorded.

[0081] S4: When loading the model, parse the computation graph of the model to be executed. For each operator node in the graph, extract its: operator unique identifier, operator type, estimated computational cost, input tensor data size, output tensor data size, as well as its direct predecessor operator ID list and direct successor operator ID list to form a complete directed data dependency graph. Calculate the data transfer amount between each pair of dependent operators as a communication strength metric.

[0082] Based on real-time task requirements, when allocating virtual modules to all operators of each required computation layer (such as an attention layer), the data dependency graph is analyzed to obtain the critical path (longest execution path) from the input to the output of that layer. The operators on the critical path are sorted according to their execution order to generate a priority mapping queue (Qcri). In the dependency graph subgraph where operators on the critical path have been removed, starting from all nodes with an in-degree of zero (i.e. no predecessor dependency), the nodes with the current in-degree of zero and their outgoing edges are repeatedly removed, and the nodes are added to the queue in turn until there are no remaining nodes, thus generating a secondary queue.

[0083] Take the first operator Ocur from the priority mapping queue, query all its predecessor operators, and find the predecessor operator Oper with the strongest communication strength with the first operator Ocur. If Ocur has already been mapped to a virtual module Vtar, then map Ocur to the same module Vtar first to achieve operator fusion and avoid data migration.

[0084] If the current load of the target module Vtar exceeds the preset load threshold Lth, then the mapping to Vtar is abandoned. Instead, in the undirected graph G, all other ready modules that are closest to Vtar's physical location (i.e., have the fewest hops) are searched, and the module Vnr with the lightest current load is selected as the mapping target of Ocur. That is, if this module is too busy, it is assigned to another module that is closest to it in physical location (with the fastest communication).

[0085] For an operator that is identified as having broadcast properties (i.e., the output of an operator is the input of multiple downstream operators), calculate the set of target virtual modules to which all direct successor operators of the operator have been mapped. Then, in the undirected graph G, find a virtual module Vctr with a central position such that the sum of the number of hops from Vctr to each module in the set is minimized, and map the broadcast operator to Vctr.

[0086] S5: For each constructed virtual module, at a fixed period, the percentage of clock cycles in which all member physical units of the virtual module are in an effective computing state is collected in real time as the computing utilization rate U. The average data buffer occupancy (in bytes) of the main output port of the virtual module is calculated as the output communication queue depth Q. The collected indicator group (U, Q) is summarized in real time to the internal database of the performance monitoring layer.

[0087] The collected indicator groups are monitored periodically. If the average utilization rate Uavg of two or more virtual modules is detected to be less than the low utilization rate threshold Ulow, the average queue depth Qinter of all data channels between the modules in the group is lower than the threshold Qlow, and this continues for Tlow monitoring cycles.

[0088] Then, module merging is triggered. First, starting from the task record of each module Vi in the inefficient module set M (|M|>2), we trace back its computation graph source to identify whether it originates from the same computation subgraph (such as different branches of an attention head).

[0089] For the identified homologous module group, the affinity score is calculated as A=γ1*(1-∑Finter / Tmon)+γ2*SPR, where γ1 and γ2 are preset weights, ∑Finter is the total amount of communication in the historical period, Tmon is the monitoring period, and SPR is the proportion of shared direct predecessors.

[0090] The combination with the highest A is selected as the candidate set to be merged, and the module Vm with the highest load is selected as the main module from the candidate set.

[0091] The operator subgraphs carried by modules Vs (excluding Vm) in the candidate set are replanned and merged into the computation flow of Vm according to the original data dependencies. This ensures that the execution order of operators within Vm after merging satisfies all data dependencies, and automatically inserts synchronization barriers when necessary.

[0092] Disband the Vs module and update the status of its member physical units in the global resource status table to idle;

[0093] Update the on-chip network routing table to redirect the data stream originally pointing to Vs to Vm;

[0094] If the computational utilization U of a single virtual module is detected to be greater than the high load threshold Uhi, the communication queue depth Q is output to be greater than the congestion threshold Qth, and this continues for Thigh monitoring cycles.

[0095] This triggers a split, analyzes the operator subgraph within a single module, identifies critical paths, and in non-critical paths, obtains candidate molecules Br that have high computational cost, whose input mainly comes from an intermediate result Rm within the module, and whose output is mainly used for subsequent calculations within the module, as the operator set.

[0096] For each Br, obtain:

[0097] Ben: The expected number of cycles to shorten the critical path of the original module after removing Br;

[0098] Cost Cos: The overhead incurred due to the need to copy intermediate results Rm to the new module and subsequent cross-module communication;

[0099] Select Br, which has the highest benefit-cost ratio Rbc=Ben / Cos, as the splitting target;

[0100] Allocate idle units from the idle resource pool, construct a new module Vn, and migrate the selected branch Br to Vn to form an independently executable task block;

[0101] After each adjustment, it enters a cooldown cycle of length Tcd, during which the adjustment is paused.

[0102] Some of the data in the above formulas are numerical calculations with dimensions removed, and the contents not described in detail in this specification are all prior art known to those skilled in the art.

[0103] The above embodiments are only used to illustrate the technical methods of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical methods of the present invention without departing from the spirit and scope of the technical methods of the present invention.

Claims

1. A multi-dimensional parallel artificial intelligence processor for large model inference, characterized in that, include: The resource management terminal is used to identify all physical computing units and obtain the undirected graph G formed by the physical communication links between units. The load decision-making end calculates the required virtual module type and the final number of virtual modules Nfn based on the batch size, sequence length, and computational layer class of the current model task. On the module construction side, all idle units are dynamically clustered into an optimized resource pool with high communication efficiency based on the undirected graph G. Then, based on the module type and Nfn, all required units are selected from the same pool for construction first. If resources are insufficient, they are supplemented from adjacent pools. The operator mapping end is used to parse the model computation graph. Based on the data dependency strength and communication cost between operators, it prioritizes mapping operators to the virtual module where their strongly dependent predecessor is located. If the module is overloaded, it will be mapped to other ready modules with the closest physical location. The performance monitoring terminal periodically collects the computing utilization and communication load of each virtual module, and triggers the merging or splitting of virtual modules based on preset thresholds and durations.

2. The multi-dimensional parallel artificial intelligence processor for large model inference according to claim 1, characterized in that, The resource management terminal is specifically used for: Each physical computing unit is divided into basic computing units and functional computing units, and each unit is assigned a unique physical identifier (PID) and type label. Record the unit type, current status, and the identifier VID of the virtual module to which each PID belongs when it is occupied, as a global resource status table; By sending a neighbor discovery request to all computing units, each computing unit reports the physical identifiers of its neighboring computing units, constructing an undirected graph G=(V,E). In the graph, each vertex v∈V represents a physical unit (identified by its PID), and each edge e(u,v)∈E indicates that there is a direct physical communication link between units u and v.

3. The multi-dimensional parallel artificial intelligence processor for large model inference according to claim 1, characterized in that, The computational layer types are attention layer and feedforward network layer; If the current layer is an attention layer, then calculate the theoretically required number of virtual attention modules Nth = ceil(Ha / Hpm) * ceil(Bs / Bpa), where Ha is the number of attention heads, Bs is the batch size, and Hpm and Bpa are preset capability parameters; If the current layer is a feedforward network layer, then the theoretical number of virtual modules required, Nth, is calculated as Nth = ceil(D / Dpm) * ceil(Bs / Bpf), where D is the hidden layer dimension, and Dpm and Bpf are preset capability parameters. Get the total number of currently idle and matching physical units Nav, and calculate the maximum buildable limit Nal = floor(Nav / Upm), where Upm is the number of core units required to build the target module. Determine the final number of builds Nbd = min(Nth, Nal), and align Nbd to a power of 2 that is no greater than its maximum power of 2 to obtain the final number of builds Nfn.

4. The multi-dimensional parallel artificial intelligence processor for large model inference according to claim 1, characterized in that, Dynamically clustering the idle units into an optimized resource pool for efficient communication includes: Each idle physical unit is used as an independent initial resource pool; Traverse all resource pools. For any two resource pools Pi and Pj, calculate their merged communication cost C = α * (average shortest path length between all cell pairs in Pi and Pj in the undirected graph G) + β * (maximum value of all shortest path lengths), where α and β are preset weighting coefficients. Select the pair with the smallest C value among all currently mergeable pools and merge them into a new resource pool. When the number of remaining resource pools decreases to the preset target value K, or the current minimum C value exceeds the preset threshold, merging stops, and K optimized resource pools are finally formed.

5. A multi-dimensional parallel artificial intelligence processor for large model inference according to claim 1, characterized in that, The operator mapping layer is specifically used for: Analyze the computational graph of the analytical model, extract operator nodes and their data dependencies, and construct a directed data dependency graph; For the computation layer that requires the allocation of virtual modules, analyze its data dependency graph, obtain the critical path from input to output, and generate a priority mapping queue Qcri according to the execution order of the operators on the critical path; In the remaining subgraph, repeatedly remove nodes with an in-degree of zero to generate a secondary queue; Prioritize mapping the current operator to the virtual module containing its predecessor operator with the strongest communication strength to achieve operator fusion; If the target module load exceeds a preset threshold, then the other ready module in the undirected graph G that is closest to the target module in terms of physical location and has the lightest load is selected for mapping.

6. A multi-dimensional parallel artificial intelligence processor for large model inference according to claim 1, characterized in that, The operator mapping layer is also used to process broadcast operators: Identify broadcast operators whose output is input to multiple downstream operators, and obtain the set of target virtual modules to which all their direct successor operators have been mapped; Find a virtual module Vctr with a central location in an undirected graph G such that the sum of the hop counts from Vctr to each module in the set is minimized, and map the broadcast operator to Vctr.

7. A multi-dimensional parallel artificial intelligence processor for large model inference according to claim 1, characterized in that, The performance monitoring layer is specifically used for: The computational utilization rate U and the output communication queue depth Q of each virtual module are collected at fixed intervals. If the average utilization of two or more virtual modules is detected to be less than the low utilization threshold, and the average queue depth of all data channels between modules is lower than the threshold Qlow, and this continues for Tlow monitoring cycles, then a module merging operation is triggered.

8. A multi-dimensional parallel artificial intelligence processor for large model inference according to claim 1, characterized in that, The module merging operation includes: Tracing back the origin of the computation graph from the inefficient module set M, we can identify whether they originate from the same computation subgraph; For the identified homologous module groups, the affinity score A = γ1*(1-∑Finter / Tmon) + γ2*SPR is calculated, where γ1 and γ2 are preset weights, ∑Finter is the total amount of communication in the historical period, Tmon is the monitoring period, and SPR is the proportion of shared direct predecessors. Select the combination with the highest affinity score as the candidate set to be merged, take the module with the highest load as the main module Vm, and merge the operator subgraphs of other candidate modules into the computation flow of the main module. Dissolve the other modules that were merged, update the status of other physical units to idle, and update the on-chip network routing table.

9. A multi-dimensional parallel artificial intelligence processor for large model inference according to claim 1, characterized in that, The performance monitoring layer is also used for: If the computational utilization U of a single virtual module is detected to be greater than the high load threshold Uhi, and the output communication queue depth Q is greater than the congestion threshold Qth, and this continues for Thigh monitoring cycles, then a module split operation is triggered.

10. A multi-dimensional parallel artificial intelligence processor for large model inference according to claim 9, characterized in that, The module splitting operation includes: Analyze the operator subgraph within this module to identify the critical path; In non-critical paths, candidate branches Br with large computational load, inputs mainly from intermediate results within the module, and outputs mainly used for subsequent calculations within the module are selected as the operator set; For each Br, obtain the revenue Ben and the cost Cos; Select Br, which has the highest benefit-cost ratio Rbc=Ben / Cos, as the splitting target; A new module is built from the idle resource pool allocation unit, and the selected branch Br is migrated to the new module.

Images