Fusion acceleration method and device based on distributed parallel strategy, equipment and medium

Abstract

The application discloses a fusion acceleration method and device based on a distributed parallel strategy, equipment and a medium. The method comprises the following steps: obtaining an operator relationship type, a communication execution type, a cluster architecture configuration and a workload feature corresponding to a target communication computing operator combination in a target fusion acceleration task; calculating a basic execution time corresponding to the target communication computing operator combination based on a preset operator type, the cluster architecture configuration and the workload feature; slicing and blocking the target communication computing operator combination based on the basic execution time to obtain a target block combination, determining a fusion acceleration strategy corresponding to the target communication computing operator combination based on the operator relationship type, the communication execution type and the target block combination; and controlling the execution order and the execution mode of a computing unit and a communication engine based on the fusion acceleration strategy through a hardware scheduler to implement the target fusion acceleration task. Through the technical scheme of the application, the hardware resource utilization rate of computation and fusion can be improved, and the model running efficiency is improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a fusion acceleration method, apparatus, device, and medium based on a distributed parallel strategy. Background Technology

[0002] With the continuous growth in the scale of Large Language Models (LLMs), the exponential increase in model parameters, sequence length, and training data volume poses a severe challenge to the computing power and storage capacity of a single computing card. Currently, distributed parallel strategies have become the mainstream paradigm for training or inferring LLMs. By combining various distributed parallel strategies, model parameters or data are split across multiple computing cards for parallel processing, thus overcoming the performance bottleneck of a single computing card.

[0003] However, the implementation of distributed parallel strategies heavily relies on aggregated communication operations between computing cards. Therefore, effectively reducing communication overhead and achieving efficient overlap and integration of computation and communication have become key technical challenges in improving the performance of LLM distributed training and inference.

[0004] In existing technologies, data partitioning and core fusion are the two main methods used to achieve computational communication overlap. However, the choice of partitioning granularity in the data partitioning method is highly dependent on the specific hardware architecture and model parameters, making it difficult to generalize the optimal partitioning method. Furthermore, multiple startups of multiple sub-cores introduce additional kernel startup overhead and context switching overhead, which may offset the performance gains from overlap. In the core fusion method, the merged core requires precise timing control and dependency management, so software scheduling overhead still exists. Moreover, communication operations are explicitly triggered and managed by the computation stream processor within the core through software, still requiring the use of streaming multiprocessor resources, and thus failing to achieve true hardware-level communication and computation offloading.

[0005] Therefore, how to improve the utilization rate of hardware resources in the convergence of computing and general computing, reduce the latency during the operation of the convergence of computing and general computing, maximize the utilization of computing and memory access efficiency, and thus improve the running efficiency of the model, is an urgent problem to be solved. Summary of the Invention

[0006] This invention provides a fusion acceleration method, device, equipment, and medium based on a distributed parallel strategy, which can solve the problems of low fusion efficiency and insufficient utilization of computing card resources in the distributed training and inference of existing large language models.

[0007] According to one aspect of the present invention, a fusion acceleration method based on a distributed parallel strategy is provided, which is applied to the model training or inference process using a distributed parallel strategy, the method comprising: Obtain the operator relationship type, communication execution type, cluster architecture configuration, and workload characteristics corresponding to the combination of target communication computation operators in the target fusion acceleration task; The base execution time corresponding to the target communication computing operator combination is calculated based on the preset operator type, the cluster architecture configuration, and the workload characteristics. The target communication computing operator combination is sliced ​​and divided into blocks based on the basic execution time to obtain the target block combination corresponding to the target communication computing operator combination. The fusion acceleration strategy corresponding to the target communication computing operator combination is determined based on the operator relationship type, communication execution type and target block combination. The hardware scheduler controls the execution order and execution mode of the computing unit and communication engine based on the fusion acceleration strategy to achieve the target fusion acceleration task.

[0008] According to another aspect of the present invention, a fusion acceleration device based on a distributed parallel strategy is provided, which is applied to a model training or inference process using a distributed parallel strategy. The device includes: The data acquisition module is used to acquire the operator relationship type, communication execution type, cluster architecture configuration and workload characteristics corresponding to the combination of target communication computing operators in the target fusion acceleration task; The time calculation module is used to calculate the basic execution time corresponding to the target communication computing operator combination based on the preset operator type, the cluster architecture configuration, and the workload characteristics. The strategy generation module is used to slice and divide the target communication computing operator combination based on the basic execution time to obtain the target block combination corresponding to the target communication computing operator combination, and to determine the fusion acceleration strategy corresponding to the target communication computing operator combination based on the operator relationship type, communication execution type and target block combination. The strategy execution module is used to control the execution order and execution mode of the computing unit and the communication engine based on the fusion acceleration strategy through the hardware scheduler, so as to realize the target fusion acceleration task.

[0009] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to execute the fusion acceleration method based on a distributed parallel strategy as described in any embodiment of the present invention.

[0010] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the fusion acceleration method based on a distributed parallel strategy as described in any embodiment of the present invention.

[0011] According to another aspect of the present invention, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the fusion acceleration method based on a distributed parallel strategy as described in any embodiment of the present invention.

[0012] The technical solution of this invention obtains the operator relationship type, communication execution type, cluster architecture configuration, and workload characteristics corresponding to the target communication computing operator combination in the target fusion acceleration task. Then, based on the preset operator type, cluster architecture configuration, and workload characteristics, the basic execution time corresponding to the target communication computing operator combination is calculated. Further, based on the basic execution time, the target communication computing operator combination is sliced ​​and divided into blocks to obtain the target block combination corresponding to the target communication computing operator combination. Then, based on the operator relationship type, communication execution type, and target block combination, the fusion acceleration strategy corresponding to the target communication computing operator combination is determined. Finally, the hardware scheduler controls the execution order and execution mode of the computing unit and communication engine based on the fusion acceleration strategy to realize the target fusion acceleration task. By taking a holistic approach from multiple levels, including system goal perception, adaptive optimization of segmentation granularity, hardware-level data path optimization, and low-overhead hardware synchronization mechanisms, communication and computation are decomposed into fine-grained blocks. The hardware maintains the dependencies between blocks, automatically handles data dependencies, and the synchronization signal mechanism between computation and communication blocks is completed by hardware. This solves the problems of low efficiency of computing fusion and insufficient utilization of computing card resources in the distributed training and inference of existing large language models. It can improve the hardware resource utilization of computing fusion, reduce the latency of computing fusion runtime, maximize the utilization of computation and memory access efficiency, and thus improve the model running efficiency.

[0013] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1This is a flowchart of a fusion acceleration method based on a distributed parallel strategy provided according to Embodiment 1 of the present invention; Figure 2 This is a schematic diagram of a large language model architecture provided in Embodiment 1 of the present invention; Figure 3 This is a schematic diagram of a hardware scheduler according to Embodiment 1 of the present invention; Figure 4 This is a flowchart of a fusion acceleration method based on a distributed parallel strategy provided according to Embodiment 2 of the present invention; Figure 5 This is a schematic diagram of a fusion acceleration strategy for equal-length slices provided in Embodiment 2 of the present invention; Figure 6 This is a schematic diagram of a fusion acceleration strategy for unequal-length slices provided in Embodiment 2 of the present invention; Figure 7 This is a schematic diagram of a fusion acceleration strategy for out-of-order scheduling provided in Embodiment 2 of the present invention; Figure 8 This is a schematic diagram of a preset communication command provided according to Embodiment 2 of the present invention; Figure 9 This is a schematic diagram of the execution flow of a hardware scheduler according to Embodiment 2 of the present invention; Figure 10 This is a flowchart of an optional fusion acceleration method based on a distributed parallel strategy provided according to Embodiment 2 of the present invention; Figure 11 This is a schematic diagram of the structure of a fusion acceleration device based on a distributed parallel strategy according to Embodiment 3 of the present invention; Figure 12 This is a schematic diagram of the structure of an electronic device that implements the fusion acceleration method based on a distributed parallel strategy according to an embodiment of the present invention. Detailed Implementation

[0016] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0017] It should be noted that the terms "first," "second," "target," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0018] Example 1 Figure 1 This is a flowchart of a fusion acceleration method based on a distributed parallel strategy provided in Embodiment 1 of the present invention. This embodiment is applicable to the fusion acceleration of model training or inference processes using a distributed parallel strategy. The method can be executed by a fusion acceleration device based on a distributed parallel strategy, which can be implemented in hardware and / or software and can be configured in an electronic device. Figure 1 As shown, the method includes: S110. Obtain the operator relationship type, communication execution type, cluster architecture configuration, and workload characteristics corresponding to the target communication computation operator combination in the target fusion acceleration task.

[0019] In this context, a large language model can refer to a deep neural network containing massive parameters and trained on massive amounts of text data. Typically, data processing or numerical computation operations performed within a single computing card are called computation operators; that is, the data processing or numerical computation operations performed by each layer or module in the large language model constitute a computation operator. Distributed parallel strategies refer to a set of strategies used during the training or inference process of a large language model to instruct the splitting of model parameters, gradients, activation values, or input data across multiple computing cards for parallel processing. Distributed parallel strategies often introduce collective communication operations between computing cards. Operations that exchange or synchronize data between multiple cards are called communication operators; that is, in a distributed parallel strategy, the operations responsible for sending data from one card to other cards or receiving data from other cards to achieve a collective communication mode are called communication operators.

[0020] Figure 2The diagram illustrates the structure of a large language model architecture provided by an embodiment of the present invention. Specifically, distributed parallel strategies can include data parallelism (DP), tensor parallelism (TP), pipeline parallelism (PP), sequence parallelism (SP), context parallelism (CP), and expert parallelism (EP), etc. Data parallelism strategies can include an all-reduce communication operator. Tensor parallelism strategies can include an all-reduce communication operator, an all-gather communication operator, and a reduce-scatter communication operator. Pipeline parallelism strategies can include a send communication operator and a receive communication operator. Sequence parallelism strategies can include an all-gather + reduce-scatter communication operator. Context parallelism strategies can include a send communication operator, a receive communication operator, an all-gather communication operator, and a reduce-scatter communication operator. Expert parallel strategies can include all-to-all communication operators. Large language models can include an input layer, a normalization layer, an attention module, an expert hybrid module, a residual addition layer, and an output layer. The attention module can include a query-key-value (QKV) linear layer, a self-attention layer, a linear layer, and a dropout layer. The expert hybrid module can include a gating layer and an expert layer.

[0021] In the large language model architecture of this invention, the input layer and normalization layer implement a sequence parallel strategy, the attention module and QKV linear layer implement a tensor parallel strategy, the attention module also implements a context parallel strategy, the dropout layer, normalization layer and gating layer implement a sequence parallel strategy, and the expert layer implements expert parallel and tensor parallel strategies. Correspondingly, a full collection communication operator is set between the normalization layer and QKV linear layer, a reduction scattering communication operator is set between the linear layer and dropout layer, a full collection communication operator is set between the QKV linear layer and self-attention layer, a full collection communication operator and a full exchange communication operator are set between the gating layer and expert layer, and a full reduction communication operator and a full exchange communication operator are set sequentially between the expert layer and the residual addition layer. It is worth noting that in this embodiment, the expert layer may include a gated projection linear layer, an up-projection linear layer, a down-projection linear layer and an activation layer. The gated projection linear layer and the up-projection linear layer are connected to the full exchange communication operator before the expert layer, and the down-projection linear layer is connected to the full reduction communication operator after the expert layer.

[0022] In this context, "fusion acceleration task" refers to a task that reduces overall execution time by overlapping or merging originally sequential computational operations with communication operations in time. The target fusion acceleration task refers to the specific fusion acceleration task that needs to be optimized. Typically, a target fusion acceleration task corresponds to one or more specific combinations of communication computation operators, which are the objects of the method execution. A combination of communication computation operators can refer to a dual unit composed of adjacent communication and computation operators. Typically, the computation and communication operators in a combination of communication computation operators are logically connected. For example, using a large language model architecture... Figure 2 Taking the example shown, the normalization layer and the first full-collection communication operator form a communication computation operator combination; the first full-collection communication operator and the QKV linear layer form a communication computation operator combination; the QKV linear layer and the second full-collection communication operator form a communication computation operator combination; the second full-collection communication operator and the self-attention layer form a communication computation operator combination; the linear layer and the reduction scattering communication operator form a communication computation operator combination; the reduction scattering communication operator and the discarding layer form a communication computation operator combination; the gated layer and the third full-collection communication operator form a communication computation operator combination; the first fully exchanged communication operator and the gated projected linear layer form a communication computation operator combination; the first fully exchanged communication operator and the up-projected linear layer form a communication computation operator combination; the down-projected linear layer and the fully reduced communication operator form a communication computation operator combination; and the second fully exchanged communication operator and the residual addition layer form a communication computation operator combination. The target communication computation operator combination can refer to the specific communication computation operator combination currently being analyzed or processed. Typically, a target communication computation operator combination can be a combination of communication computation operators that are adjacent to computation operators and have data dependencies. It is worth noting that if there are three consecutive operators such as "computation-communication-computation" or "communication-computation-communication", then these three consecutive operators need to be considered as a single target communication computation operator combination. For example, continuing from the above example, in this embodiment of the invention, the first full collection communication operator and the QKV linear layer constitute a target communication computation operator combination; the linear layer, the reduction scattering communication operator, and the discard layer constitute a target communication computation operator combination; the first full exchange communication operator, the gated projection linear layer, and the upper projection linear layer constitute a target communication computation operator combination; the lower projection linear layer and the full reduction communication operator constitute a target communication computation operator combination; and the second full exchange communication operator and the residual addition layer constitute a target communication computation operator combination.

[0023] The operator relationship type can refer to the type used to describe the direction of data dependency between communication operators and computation operators in a combination of communication and computation operators. For example, the operator relationship type can be either computation-before-communication or communication-before-computation. Typically, in the computation-before-communication type, the output of the computation operator is the input of the communication operator, and in the communication-before-computation type, the output of the communication operator is the input of the computation operator. The communication execution type can refer to the type used to describe the constraint characteristics of the communication operator itself on the block execution order. For example, the communication execution type can be either order-guaranteed or out-of-order. Typically, the order-guaranteed type indicates that the communication result is related to the block execution order, i.e., it must be processed according to the block index order. The out-of-order type indicates that the communication result is independent of the block execution order, i.e., out-of-order block execution is allowed. The cluster architecture can refer to the organization and topology of the entire hardware system used for training or inference of large language models. For example, the cluster architecture can include the connection method between multiple computing cards within a server node, or the connection method between different server nodes. The cluster architecture configuration can refer to the set of parameters used to describe the capabilities of the distributed cluster hardware. For example, cluster architecture configurations can include single-computing-card configurations and cluster interconnect configurations. Workload characteristics can refer to characteristic parameters used to describe the amount of data and computation required to process the target communication computation operator combination. Typically, workload characteristics are determined by model hyperparameters and parallel strategies. For example, workload characteristics can include the amount of communication data and computational load.

[0024] S120. Calculate the basic execution time corresponding to the target communication computing operator combination based on the preset operator type, the cluster architecture configuration, and the workload characteristics.

[0025] The preset operator type can refer to a predefined operator category used to distinguish between computation operators and communication operators. For example, the preset operator type can be either a computation operator type or a communication operator type. Execution time can refer to the time taken for a single communication computation operator combination to complete execution. Base execution time can refer to the estimated execution time of a single communication computation operator combination based on the cluster architecture configuration and workload characteristics, without chunking optimization.

[0026] S130. Based on the basic execution time, the target communication computing operator combination is sliced ​​and divided into blocks to obtain the target block combination corresponding to the target communication computing operator combination. Based on the operator relationship type, communication execution type and target block combination, the fusion acceleration strategy corresponding to the target communication computing operator combination is determined.

[0027] Slicing and partitioning refer to the operation of dividing a originally continuous computation or communication task into multiple independent subtasks. Typically, each subtask is called a block. Block combination refers to the set of blocks corresponding to the same slicing and partitioning strategy. Generally, multiple block combinations can correspond to the same target communication computation operator combination. Target block combination refers to the set of optimal block sizes determined after optimization search. Each optimal block size represents the optimal block execution time. Fusion acceleration strategy refers to a specific scheduling scheme based on the operator relationship type, communication execution type, and target block combination of the target communication computation operator combination. Typically, fusion acceleration strategy can include block execution order, pipeline overlap mode, and resource allocation strategy.

[0028] S140. The hardware scheduler controls the execution order and execution mode of the computing unit and the communication engine based on the fusion acceleration strategy to achieve the target fusion acceleration task.

[0029] The hardware scheduler refers to a hardware module used to dynamically control the execution order and method of computing units and communication engines according to the convergence acceleration strategy. Typically, the hardware scheduler is located inside the computing card or tightly coupled to it. Figure 3 The diagram illustrates the structure of a hardware scheduler according to an embodiment of the present invention. Specifically, the hardware scheduler includes a chunk queue, a dependency tracker, a resource allocator, and a resource scheduler. The chunk queue can refer to a First-In-First-Out (FIFO) queue used to cache pending chunk tasks and their metadata. The dependency tracker can refer to a module used to maintain data dependencies between chunks and perform dependency checks to determine whether a chunk meets the execution conditions. The resource allocator can refer to a module used to allocate specific hardware execution resources to ready chunks that have passed dependency checks. The resource scheduler can refer to a module used to determine the actual launch order and specific execution method of ready chunks.

[0030] The computing unit can refer to the hardware module that performs computational operations. For example, a computing unit can include a Streaming Multiprocessor (SM), Tensor Core, and Compute Unified Device Architecture (CUDA) core within a Graphics Processing Unit (GPU). The communication engine can refer to the hardware module that performs communication operations. For example, a communication engine can include a network interface, a Direct Memory Access (DMA) engine, a network interface card (NIC) buffer, and a remote memory access path. The execution order can refer to the order in which the various blocks are scheduled for execution on the hardware. The execution method can refer to the specific mode of block execution. For example, the execution method can include: whether the computational and communication blocks are executed serially or in parallel; whether the communication blocks are directly written to remote memory via stored instructions or indirectly transmitted via memory copy; and whether the synchronization mechanism uses hardware bitmaps and wait queues or software memory barriers.

[0031] The technical solution of this invention obtains the operator relationship type, communication execution type, cluster architecture configuration, and workload characteristics corresponding to the target communication computing operator combination in the target fusion acceleration task. Then, based on the preset operator type, cluster architecture configuration, and workload characteristics, the basic execution time corresponding to the target communication computing operator combination is calculated. Further, based on the basic execution time, the target communication computing operator combination is sliced ​​and divided into blocks to obtain the target block combination corresponding to the target communication computing operator combination. Then, based on the operator relationship type, communication execution type, and target block combination, the fusion acceleration strategy corresponding to the target communication computing operator combination is determined. Finally, the hardware scheduler controls the execution order and execution mode of the computing unit and communication engine based on the fusion acceleration strategy to realize the target fusion acceleration task. By taking a holistic approach from multiple levels, including system goal perception, adaptive optimization of segmentation granularity, hardware-level data path optimization, and low-overhead hardware synchronization mechanisms, communication and computation are decomposed into fine-grained blocks. The hardware maintains the dependencies between blocks, automatically handles data dependencies, and the synchronization signal mechanism between computation and communication blocks is completed by hardware. This solves the problems of low efficiency of computing fusion and insufficient utilization of computing card resources in the distributed training and inference of existing large language models. It can improve the hardware resource utilization of computing fusion, reduce the latency of computing fusion runtime, maximize the utilization of computation and memory access efficiency, and thus improve the model running efficiency.

[0032] Example 2 Figure 4This is a flowchart of a fusion acceleration method based on a distributed parallel strategy provided in Embodiment 2 of the present invention. This embodiment is a refinement based on the above embodiment. Specifically, this embodiment refines the step of "calculating the basic execution time corresponding to the target communication computing operator combination based on the preset operator type, the cluster architecture configuration, and workload characteristics." Specifically, it may include: if the preset operator type is a computing operator type, then obtaining the peak computing power and computing efficiency of a single card in the cluster architecture configuration, calculating the computing volume corresponding to the target computing operator in the target communication computing operator combination based on the workload characteristics, and determining the first execution time corresponding to the target computing operator based on the peak computing power, computing efficiency, and computing volume of the single card; if the preset operator type is a communication operator type, then obtaining the cluster interconnect bandwidth and communication efficiency in the cluster architecture configuration, calculating the data volume corresponding to the target communication operator in the target communication computing operator combination based on the workload characteristics, and determining the second execution time corresponding to the target communication operator based on the cluster interconnect bandwidth, communication efficiency, and data volume; combining the first execution time and the second execution time as the basic execution time corresponding to the target communication computing operator combination. Figure 4 As shown, the method includes: S210. Obtain the target model architecture, target distributed parallel strategy, and target cluster architecture corresponding to the target fusion acceleration task.

[0033] Here, "model architecture" refers to the network structure design of the large language model itself. For example, model architecture can include the number of layers, hidden dimensions, number of attention heads, intermediate dimensions of the feedforward network, activation function type, and whether hybrid experts are used. Typically, model architecture determines the types of computational operators and tensor shapes. "Target model architecture" refers to the architecture of the specific large language model currently being processed. "Target distributed parallel strategy" refers to the combination of parallel strategies adopted for the target model architecture. "Target cluster architecture" refers to the topology and connection method of the actual hardware cluster currently deployed for training or inference tasks.

[0034] S220. Based on the target model architecture and the target distributed parallel strategy, determine the target communication computing operator combination corresponding to the target fusion acceleration task and the operator relationship type corresponding to the target communication computing operator combination.

[0035] The target communication computation operator combination includes at least one target computation operator and one target communication operator. The target computation operator can refer to the operator that performs the computational task in the target communication computation operator combination. For example, the target computation operator can be matrix multiplication, activation functions, and normalization, etc. The target communication operator can refer to the operator that performs the communication task in the target communication computation operator combination.

[0036] S230. Based on the preset communication library specifications, perform semantic analysis on the target distributed parallel strategy to determine the communication execution type corresponding to the combination of target communication computation operators.

[0037] The predefined communication library specification refers to a standard behavioral description of a predefined collection communication library. Typically, the predefined communication library specification explicitly defines the semantics of each communication operator. For example, the predefined communication library specification could be: the result of AllReduce is related to the block order and belongs to the order-guaranteed type. Semantic analysis refers to the process of inferring the communication execution type based on the predefined communication library specification and the communication pattern in the target distributed parallel strategy.

[0038] S240. Determine the cluster architecture configuration corresponding to the target cluster architecture based on the hardware configuration file, and calculate the workload characteristics corresponding to the target communication computing operator combination based on the model hyperparameters corresponding to the target model architecture and the target distributed parallel strategy.

[0039] In this context, the hardware configuration file can refer to a static file used to store specific parameters of the target cluster architecture. Model hyperparameters can refer to configurable numerical parameters of the model architecture or model runtime. For example, model hyperparameters can include hidden dimensions, sequence length, micro-batch size, number of attention heads, and number of layers.

[0040] Specifically, the process begins by reading the user-specified target model architecture and target distributed parallel strategy, and then determining the target cluster architecture based on task deployment information. Next, the model computation graph containing the target model architecture and target distributed parallel strategy is parsed, identifying all communication and computation operator pairs with data dependencies. This forms the target communication-computation operator combination, and the operator relationship type is determined: if the input of a communication operator comes from the output of a computation operator, then computation precedes communication; otherwise, communication precedes computation. Afterward, semantic analysis of the model computation graph is performed according to the preset communication library specifications to determine the communication execution type. The cluster architecture configuration is then directly read from the hardware configuration file. Finally, the workload characteristics of each target communication-computation operator combination are calculated based on the model hyperparameters and the target distributed parallel strategy. Thus, by comprehensively acquiring information from three dimensions—model architecture, parallel strategy, and cluster architecture—one-sided optimization relying on only a single factor can be avoided. Semantic analysis using the preset communication library specifications ensures that the determination of the communication execution type is deterministic and reproducible, independent of runtime trial and error, providing reliable input for subsequent segmentation and optimization.

[0041] S250. If the preset operator type is a computing operator type, then obtain the single-card peak computing power and computing efficiency in the cluster architecture configuration, calculate the computing volume corresponding to the target computing operator in the target communication computing operator combination based on the workload characteristics, and determine the first execution time corresponding to the target computing operator based on the single-card peak computing power, computing efficiency and computing volume.

[0042] In this context, peak computing power per card refers to the number of floating-point operations per second that a single computing card can perform under ideal conditions. Computational efficiency refers to the ratio of actual computation time to theoretical computation time. For example, computational efficiency = actual achieved computing power / peak computing power per card. Typically, computational efficiency is less than 1. Computational load refers to the total number of floating-point operations required for the target computing operator to execute. First execution time refers to the estimated execution time of the target computing operator without sharding. For example, first execution time can be calculated using the formula: First execution time = computational load / (peak computing power per card × computational efficiency).

[0043] In an optional implementation, calculating the computational load corresponding to the target computational operator in the target communication computational operator combination based on workload characteristics includes: Step a1: Based on the target computing operator in the target communication computing operator combination, determine the corresponding first feature parameter set in the workload characteristics.

[0044] Among these, feature parameters can refer to the basic quantifiable variables that constitute the characteristics of the workload. For example, feature parameters can include micro-batch size (b), sequence length (s), hidden dimension (h), tensor parallelism (t), number of attention heads, and number of layers, etc. The first feature parameter can refer to the feature parameter used to calculate the computational load of the target computational operator. The set of first feature parameters can refer to the set of all first feature parameters corresponding to the same target computational operator. Typically, the set of first feature parameters may differ for different target computational operators. For example, taking the target model architecture and target distributed parallel strategy as an example... Figure 2 For example, if the target computation operator is the QKV linear layer in the target communication computation operator combination consisting of the first full collection communication operator and the QKV linear layer, then the first feature parameter set can be represented as: {2,3,b,s,h} 2 If the target computation operator is the linear layer and the dropout layer in the combination of the target communication computation operator consisting of the linear layer, the reduced scattering communication operator, and the dropout layer, then the set of the first feature parameters corresponding to the linear layer can be expressed as: {2, b, s, h}. 2The discard layer contains one-dimensional (1D) element-wise computation and Special Function Unit (SFU) computation. The first feature parameter set corresponding to the 1D element-wise computation can be represented as: {4,b,s,h,1 / t}; the first feature parameter set corresponding to the SFU computation can be represented as: {b,s,h,1 / t}. The computation operator corresponding to the linear layer is a 2D operation computation operator. If the target computation operator is the gated projection linear layer and the upprojection linear layer in the target communication computation operator combination consisting of the fully switched communication operator, the gated projection linear layer, and the upprojection linear layer, then the first feature parameter set can be represented as: {4,b,s,h,1 / t,h}. ff}. h ff This can represent the expert intermediate dimension. If the target computation operator is the downprojective linear layer in the target communication computation operator combination consisting of the downprojective linear layer and the fully reduced communication operator, then the first feature parameter set can be represented as: {2,b,s,h,1 / t,h} ff}

[0045] Step a2: Based on the product of each first feature parameter in the first feature parameter set, determine the computational amount corresponding to the target computation operator in the target communication computation operator combination.

[0046] For example, continuing from the above example, if the target computation operator is the QKV linear layer in the target communication computation operator combination consisting of the first full collection communication operator and the QKV linear layer, then the computational complexity can be expressed as: If the target computation operator is a linear layer and a dropout layer in a combination of target communication computation operators consisting of a linear layer, a reduced scattering communication operator, and a dropout layer, then the computational complexity corresponding to the 1D element-wise operation computation in the dropout layer can be expressed as: The computational cost of SFU computation in the dropout layer can be expressed as: The computational complexity of 2D operations in a linear layer can be expressed as: If the target computation operator is the gated projective linear layer and the upprojective linear layer in the target communication computation operator combination consisting of a fully switched communication operator, a gated projective linear layer, and an upprojective linear layer, then the total computational complexity can be expressed as: The computational complexity of the gated projection linear layer can be expressed as: The computational cost of the upward projection linear layer can be expressed as: If the target computation operator is the downprojective linear layer in the target communication computation operator combination consisting of the downprojective linear layer and the fully reduced communication operator, then the computational complexity can be expressed as: .

[0047] Specifically, for each computational operator in the target communication computational operator combination, the first characteristic parameter set corresponding to the operator type is retrieved from a predefined computational load template library. Then, the current model hyperparameters and parallel strategy values ​​are read from the workload features. Next, these values ​​are substituted into the corresponding positions in the set, and continuous multiplication operations are performed to finally obtain the computational load. Thus, by using the parameter set product method to perform unified paradigm quantization of computational operators, the time estimation module achieves high consistency and scalability, providing a solid foundation for subsequent operations.

[0048] It is worth noting that, in this embodiment of the invention, if the same target communication computation operator combination contains multiple computation operators of different types of computation units, such as computation operators in tensor computation units, computation operators in special function computation units, or computation operators in vector computation units, then these multiple computation operators of different types of computation units can be respectively used as target computation operators corresponding to the target communication computation operator combination, and then the computational amount and first execution time corresponding to the target computation operator can be calculated. For example, if the target communication computation operator combination contains a linear layer, a reduction scattering communication operator, and a dropout layer, then the linear layer and the dropout layer can be respectively used as target computation operators. Furthermore, if the number of computational operations in the same target computation operator is multiple, then the data volume and execution time of each computational operation can be calculated separately, and then the execution time of the target computation operator can be determined according to the execution order of each computational operation in the target computation operator. For example, taking the target computation operator as the discard layer, the target computation operator includes 1D element-wise operation computation and SFU computation. Then, the computational amount and execution time corresponding to the 1D element-wise operation computation and the computational amount and execution time corresponding to the SFU computation can be calculated respectively. Then, based on the execution order of each computation operation in the discard layer, the execution time of the 1D element-wise operation computation and the execution time of the SFU computation, the first execution time corresponding to the discard layer can be determined.

[0049] S260. If the preset operator type is a communication operator type, then obtain the cluster interconnection bandwidth and communication efficiency in the cluster architecture configuration, calculate the data volume corresponding to the target communication operator in the target communication computing operator combination based on the workload characteristics, and determine the second execution time corresponding to the target communication operator based on the cluster interconnection bandwidth, communication efficiency and data volume.

[0050] Here, cluster interconnect bandwidth refers to the maximum data transfer rate of the physical link for data transmission between computing cards. Communication efficiency refers to the ratio of actual communication bandwidth to theoretical cluster interconnect bandwidth. For example, communication efficiency = actual communication bandwidth / cluster interconnect bandwidth. Typically, communication efficiency is strongly correlated with message size; efficiency is low for small messages and approaches 1 for large messages. Data volume refers to the total number of bytes that the target communication operator needs to transmit. Second execution time refers to the estimated execution time of the target communication operator without segmentation. For example, second execution time can be calculated using the formula: Second execution time = Data volume / (Cluster interconnect bandwidth × Communication efficiency).

[0051] In an optional implementation, calculating the data volume corresponding to the target communication operator in the target communication computing operator combination based on workload characteristics includes: Step b1: Based on the target communication operator in the target communication computation operator combination, determine the corresponding second feature parameter set in the workload characteristics.

[0052] The second characteristic parameter can refer to the characteristic parameter used to calculate the communication data volume of the communication operator. The set of second characteristic parameters can refer to the set of all the second characteristic parameters corresponding to the same target communication operator. Typically, the set of second characteristic parameters corresponding to different target communication operators may be different. For example, continuing from the above example, if the target communication operator is the first full-collection communication operator in the combination of the first full-collection communication operator and the QKV linear layer, then the set of second characteristic parameters can be expressed as: {2,b,s,h,(t-1) / t}. If the target communication operator is the reduction-scattering communication operator in the combination of the linear layer, the reduction-scattering communication operator, and the discarding layer, then the set of second characteristic parameters can be expressed as: {2,b,s,h,(t-1) / t}. If the target communication operator is the fully switched communication operator in the combination of the fully switched communication operator, the gated projective linear layer, and the up-projective linear layer, then the second set of feature parameters can be expressed as: {2,b,s,h,(e-1) / e,K}. e can represent the expert parallelism, and K can represent the number of candidates selected based on the Top K expert selection strategy. If the target communication operator is the fully reduced communication operator in the combination of the down-projective linear layer and the fully reduced communication operator, then the second set of feature parameters can be expressed as: {4,b,s,h,K,(t-1) / t}.

[0053] Step b2: Based on the product of each second feature parameter in the second feature parameter set, determine the data volume corresponding to the target communication operator in the target communication computation operator combination.

[0054] For example, continuing from the above example, if the target communication operator is the first full-collection communication operator in the target communication computation operator combination consisting of the first full-collection communication operator and the QKV linear layer, or if the target communication operator is the reduced-scattering communication operator in the target communication computation operator combination consisting of the linear layer, the reduced-scattering communication operator, and the discarding layer, then the data volume can be expressed as: If the target communication operator is the fully switched communication operator in the combination of the fully switched communication operator, the gated projective linear layer, and the upprojective linear layer, then the data volume can be expressed as: If the target communication operator is the fully reduced communication operator in the target communication computation operator combination consisting of the lower projected linear layer and the fully reduced communication operator, then the data volume can be expressed as: .

[0055] Specifically, for communication operators in a combination of communication computation operators targeting the same goal, the corresponding set of second feature parameters is first retrieved from a predefined data volume template library. Then, the same parameter values ​​from the workload features are read and multiplied sequentially to obtain the final data volume. Thus, by employing a parameter set multiplication method to perform unified paradigm quantization of communication operators, the time estimation module achieves high consistency and scalability, providing a solid foundation for subsequent operations.

[0056] S270. Combine the first execution time and the second execution time to form the base execution time corresponding to the target communication computing operator combination.

[0057] Specifically, for the target communication computation operator combination, firstly, the type of the currently processed operator is determined: if it is a computation operator, the peak computing power of a single computing card of that model is read from the cluster architecture configuration, and the corresponding computational efficiency is queried based on the type of computation operator and the shape of the input tensor using a pre-running micro-benchmark table. Then, based on the model hyperparameters (b, s, h) and tensor parallelism t, the total number of floating-point operations (equivalent to computational load) of the computation operator is calculated, and substituted into the formula: First execution time = Computational load / (Peak computing power × Computational efficiency) to obtain the computation time, i.e., the first execution time. If the current operator is a communication operator, the cluster interconnect bandwidth and corresponding communication efficiency are obtained from the hardware configuration, and the amount of communication data is calculated based on the parallel strategy and model hyperparameters, substituted into the formula: Second execution time = Data amount / (Bandwidth × Communication efficiency) to obtain the communication time, i.e., the second execution time. Finally, the first execution time and the second execution time are combined, either by addition or by taking the maximum value, as the base execution time of the combination, used as a benchmark reference for subsequent block optimization.

[0058] S280. The target communication computing operator combination is sliced ​​and divided into blocks based on a preset candidate space to determine the candidate block set corresponding to the target communication computing operator combination.

[0059] The preset candidate space can refer to a predefined set of optional segmentation methods. For example, the preset candidate space can include a range of the number of blocks, a range of the size of each block, and whether the block sizes are different. The range of the number of blocks can represent the number of blocks allowed to be segmented. The range of the size of each block can restrict the size of each block to be greater than a minimum granularity threshold, and the sum of the sizes of all blocks equals the total sequence length. It is worth noting that in this embodiment of the invention, the communication operators and computation operators in the same target communication computation operator combination can be required to use the same segmentation scheme in the preset candidate space, or different segmentation schemes can be matched for the communication operators and computation operators in the same target communication computation operator combination. This embodiment of the invention does not specifically limit this. A candidate block combination can refer to a specific segmentation scheme in the candidate space. Typically, a candidate block combination can include the block size and block order corresponding to the segmentation scheme. For example, let's consider a preset candidate space with the following parameters: the number of blocks ranges from X=4; the size of each block is not less than 256, and the sum of all block sizes equals 2048; allowing for non-uniform block sizes. Then, the candidate block combination could be {s1=256, s2=512, s3=512, s4=768}, s i This can represent the i-th block under the segmentation scheme, where i = 1, 2, 3, 4. The candidate block set can refer to the set of multiple candidate block combinations corresponding to the same target communication computation operator combination. The candidate block set contains at least one candidate block combination, and each candidate block combination contains at least one computation block and its corresponding communication block.

[0060] S290. Analyze the execution time of each candidate block combination based on the preset search algorithm and the basic execution time, and determine the target block combination corresponding to the target communication computing operator combination based on the time analysis results.

[0061] Here, the preset search algorithm can refer to an optimization algorithm used to find the optimal block combination in the candidate space. Time analysis can refer to the process of calculating the overall execution time under the candidate block combination. The time analysis result can refer to the overall execution time value corresponding to the candidate block combination obtained after time analysis.

[0062] Specifically, firstly, a preset candidate space is constructed based on the total sequence length *s* and the minimum granularity threshold, thereby generating a finite number of candidate block combinations, resulting in a candidate block set. Then, for each candidate block combination, the computation and communication times corresponding to each block are calculated using a time estimation formula. Next, an execution time analysis is performed on each candidate block combination using a preset search algorithm and a baseline execution time. The combination with the smallest time analysis result, which is less than the baseline execution time, is determined as the target block combination and output to the subsequent hardware scheduler.

[0063] In an optional implementation, execution time analysis is performed on each candidate block combination based on a preset search algorithm and basic execution time, and the target block combination corresponding to the target communication computation operator combination is determined based on the time analysis results, including: Step c1: Determine the total overlap time model corresponding to the candidate block combination based on the operator relationship type and communication execution type corresponding to the target communication calculation operator combination, and determine the basic total overlap time corresponding to the candidate block combination based on the total overlap time model.

[0064] The total overlap time refers to the total time taken from the start of execution of the first block to the completion of the last block when using a pipelined overlapping execution method, i.e., computation and communication are performed in parallel. The total overlap time model can be a mathematical expression predetermined based on the operator relation type and communication execution type. Typically, the total overlap time model can be used to calculate the total overlap time for a given block combination. It is worth noting that in this embodiment of the invention, computation blocks and communication blocks with the same block order within the same candidate block combination can be treated as a single block object. For example, the computation block with block order i... Communication blocks in the corresponding block order As a whole, the chunked object. For example, if the operator relation type is: computation before communication, and the communication execution type is: sequence-guaranteed, then the overlapping total time model can be represented as: . This can represent the execution time of a single computation block, that is, the time required for a computation operator to complete from start to finish within a block size. Typically, The calculation process is consistent with the calculation process of the first execution time. This can represent the execution time of the first computation block, i.e. . It can represent the execution time of the (i+1)th computation block. This can represent the execution time of a single communication block, that is, the time required for the communication operator to complete from start to finish within a block size. Typically, The calculation process for the first execution time is consistent with the calculation process for the second execution time. It can represent the execution time of the i-th communication block. This can represent the execution time of the last communication block, i.e. If the operator relation type is: communication before computation, and the communication execution type is: sequential guarantee type, then the overlapping total time model can be expressed as: If the communication execution type is out-of-order, then the total overlap time model can be expressed as: The base overlap total time refers to the value obtained by substituting the computation and communication times of each block into the overlap total time model. Typically, the base overlap total time is the estimated total execution time of the candidate block combination under ideal pipeline overlap.

[0065] Step c2: Analyze the execution time of each basic overlap total time based on the basic execution time, determine the target overlap total time that satisfies the basic execution time, and take the candidate block combination corresponding to the target overlap total time as the target block combination corresponding to the target communication computing operator combination.

[0066] The target total overlap time can refer to the optimal total time selected through execution time analysis among the base total overlap times corresponding to all candidate block combinations. For example, the target total overlap time can be the time value that minimizes the base total overlap time without exceeding the base execution time.

[0067] Specifically, for each candidate block combination in the candidate block set, the corresponding total overlap time model can be selected based on the operator relationship type and communication execution type of the target communication calculation operator combination. Then, using the previously calculated computation time and communication time of each block, the basic total overlap time of the combination is calculated by substituting it into the model. Next, this value is compared with the basic execution time calculated in the previous steps. The above calculation process is repeated for all candidate combinations, and the time with the smallest basic total overlap time that does not exceed the basic execution time is selected as the target total overlap time. The candidate block combination corresponding to this time is then determined as the target block combination. It is worth noting that if the basic total overlap time of a candidate block combination is greater than the basic execution time, it is directly discarded and not included in the selection. Thus, by replacing actual running tests with mathematical calculations, the search cost can be significantly reduced. At the same time, using the basic execution time as a reference can ensure positive benefits from the optimization results, providing a valid basis for subsequent operations.

[0068] S2100. Based on the operator relationship type, communication execution type, and target block combination, determine the fusion acceleration strategy corresponding to the target communication computing operator combination.

[0069] Specifically, after determining the target block combination, the operator relationship type and communication execution type pre-determined for that target communication computation operator combination can be read. If the communication execution type is sequentially guaranteed, the fusion acceleration strategy stipulates that each block must be executed strictly according to the block index order. That is, the computation and communication of the first block are processed first, and the order of computation or communication is determined according to the operator relationship type. Then, subsequent blocks are processed in sequence, and adjacent blocks adopt a pipelined overlapping mode: in the scenario of computation before communication, the corresponding communication block is started immediately after each computation block is completed, and the next computation block begins at the same time; in the scenario of communication before computation, the corresponding computation block is started immediately after each communication block is completed, and the next communication block begins at the same time. If the communication execution type is out-of-order, the fusion acceleration strategy allows out-of-order block scheduling. That is, computation blocks and communication blocks can be launched independently in any order of readiness, as long as their corresponding block sizes match. The system does not need to maintain the block index order, but only needs to ensure the data dependency relationship between each computation block and its corresponding communication block. Ultimately, the convergence acceleration strategy is encapsulated into a set of hardware-recognizable configuration parameters, including: a list of chunk sizes, the start conditions for each chunk (i.e., a dependency table), chunk scheduling order flags (such as sequential or out-of-order), and pipeline direction flags (such as compute priority or communication priority), which are then passed to the hardware scheduler for execution.

[0070] Figure 5 The diagram illustrates a fusion acceleration strategy for equal-length slice blocks provided in an embodiment of the present invention. Specifically, computation block 1 and communication block 1 form the first block object, computation block 2 and communication block 2 form the second block object, and computation block 3 and communication block 3 form the third block object. The execution time of each computation block is T1, and the execution time of each communication block is T2. There is a dependency between computation block 1 and communication block 1. There is a dependency between computation block 2 and communication block 2. There is a dependency between computation block 3 and communication block 3. The first block object, the second block object, and the third block object are executed sequentially.

[0071] Figure 6The diagram illustrates a fusion acceleration strategy for unequal-length slices provided in an embodiment of the present invention. Specifically, computation block 1 and communication block 1 form the first block object, computation block 2 and communication block 2 form the second block object, and computation block 3 and communication block 3 form the third block object. The execution time of computation block 1 is T1, the execution time of computation block 2 is T2, the execution time of computation block 3 is T3, the execution time of communication block 1 is T4, the execution time of communication block 2 is T5, and the execution time of communication block 3 is T6. There is a dependency between computation block 1 and communication block 1. There is a dependency between computation block 2 and communication block 2. There is a dependency between computation block 3 and communication block 3. The first block object, the second block object, and the third block object are executed sequentially.

[0072] Figure 7 The diagram illustrates a fusion acceleration strategy with out-of-order scheduling provided by an embodiment of the present invention. Specifically, computation block 1 and communication block 1 form the first block object, computation block 2 and communication block 2 form the second block object, and computation block 3 and communication block 3 form the third block object. The execution time of computation block 1 is T1, the execution time of computation block 2 is T2, the execution time of computation block 3 is T3, the execution time of communication block 1 is T4, the execution time of communication block 2 is T5, and the execution time of communication block 3 is T6. There is a dependency between computation block 1 and communication block 1. There is a dependency between computation block 2 and communication block 2. There is a dependency between computation block 3 and communication block 3. The first block object, the second block object, and the third block object are executed out of order. For example, the second block object is executed first, then the first block object, and finally the third block object.

[0073] S2110. The fusion acceleration strategy is parsed and processed through a block queue to obtain the communication computing dependency relationship corresponding to the fusion acceleration strategy.

[0074] In this context, communication-computation dependency refers to the sequential constraints between computation blocks and communication blocks within each block object in a convergence acceleration strategy, arising from the data flow. For example, if the operator relationship type is computation before communication, the input data of the communication block depends on the output result of the corresponding computation block; that is, the communication block must wait for its corresponding computation block to complete before it can start. If it is communication before computation, the computation block depends on the data received by the corresponding communication block; that is, the computation block must wait for the corresponding communication block to complete before it can execute. The communication execution type further affects the granularity of the dependency: under the sequential guarantee type, the block index order also constitutes an implicit dependency, such as the (i+1)th block depending on the completion of the ith block; under the out-of-order type, only direct data dependencies exist between corresponding blocks, and there are no sequential constraints between blocks with different indices.

[0075] S2120. The dependency check is performed on the communication computation dependency based on the target block combination and operator relationship type by the dependency tracker to obtain the dependency check result.

[0076] Dependency checking refers to the process by which the dependency tracker performs real-time judgments on the pre-dependency conditions of each block based on the target block combination and operator relationship type. Typically, for a block to be executed, the dependency tracker queries all its dependencies, such as whether the dependent computation results are ready, whether the dependent communication data has been written to local memory, etc., and whether all are satisfied. If satisfied, the block is marked as ready; if not, the block is suspended, waiting for a dependency completion signal. It is worth noting that in this embodiment of the invention, dependency checking is performed in hardware parallel mode, without software polling. The dependency checking result can refer to the set of status information output after the dependency checking. For example, the dependency checking result may include: the ready flag of each block, the identifier of unsatisfied dependencies, the priority of the ready block, and the dependency graph update signal. Typically, this dependency checking result is passed to the resource allocator and resource scheduler to determine which hardware resources to allocate next and which ready block to schedule for execution.

[0077] S2130. The resource allocator determines the computing unit and communication engine corresponding to the target block combination in the fusion acceleration strategy based on the dependency check results.

[0078] S2140. The execution order and execution mode of the computing unit and the communication engine are controlled by the resource scheduler based on the preset synchronization mechanism and preset communication instructions to achieve the target fusion acceleration task.

[0079] The preset synchronization mechanism is a hardware-maintained bitmap and waiting queue synchronization mechanism; the preset communication instructions include storage instructions and loading instructions; the storage instructions are used in scenarios where the communication flow depends on the computing flow, and data is directly written to the remote computing card memory through unified memory addressing mapping; the loading instructions are used in scenarios where the computing flow depends on the communication flow, and data is directly read from the remote computing card memory through unified memory addressing mapping; when the computing unit executes the storage instructions or loading instructions, it directly triggers network transmission, bypassing memory copying and reducing the consumption of streaming processor resources.

[0080] The store instruction reads a contiguous file of vector registers with identifiers (IDs) of 2N and 2N+1 specified in the Vector Register Table (VRT), forming a 64-bit vector base address. This base address is then added to a 32-bit offset address in the register specified in the Scalar Register Source (SRS), resulting in a 64-bit memory access address. The memory access unit reads the vector register specified in the Vector Register Destination (VRD), using the address region specified by the type (TYPE) of the memory address region, and writes the number of threads (WARP_SIZE) contained in the thread bundle from the VRD into the specified memory location. Each data element is SIZE bytes in size, and the memory location type includes remote memory. Specifically, the instruction format of the storage instruction can be shown in Table 1 below. In the storage instruction, bits [31:24] represent the VRD vector register, which is used to store the vector data to be written; bits

[23] and [8] together constitute the TYPE field, which is used to describe the storage type; bits [22:16] represent the 32-bit offset address specified by SRS; bits [15:9] represent the VRT vector register, which is used to store the 64-bit vector base address; bit [7] is a constant bit; bits [6:2] represent the instruction code identifier; bits [1:0] represent the size of each data element. TYPE[0] represents local memory; TYPE[1] represents shared memory; TYPE[2] represents local global memory; TYPE[3] represents remote memory. The SIZE size can be 0, 1 or 2. SIZE=0 represents a word with 32-bit binary digits (Bit); SIZE=1 represents a half-word with 16 bits; SIZE=2 represents a signed character with 8 bits.

[0081] Table 1. Instruction format for store instructions The load instruction reads the contiguous vector register file with register IDs 2N and 2N+1 specified by VRT, forming a 64-bit vector base address. This base address is then added to the 32-bit offset address in the register specified by SRS, forming a 64-bit memory access address. The memory access unit uses an independent vector address to access the address region specified by TYPE, obtaining WARP_SIZE data points specified by SIZE, and writing them back to the corresponding channel of the vector register specified by VRD. The memory unit type includes remote memory. Specifically, the instruction format for the load instruction can be as shown in Table 2 below, where SIZE can be 0, 1, 2, or 3. SIZE=3 indicates that the data is an 8-bit unsigned character.

[0082] Table 2. Loading instruction format Figure 8 The diagram illustrates a preset communication command provided by an embodiment of the present invention. Specifically, the buffer on the computing card is directly connected to the network interface, and the computing unit's load or store command directly triggers network data transmission. After the computing unit completes its calculation, the result stored in register L0 or shared memory L1 can be directly sent to remote memory via a store command, without going through the local computing card's global memory L3. Similarly, before the computing unit starts its calculation, data stored in remote memory can be directly loaded into register L0 or L1 via a load command, without going through the local computing card's global memory L3. Thus, through unified memory addressing mapping, remote memory can be accessed as if it were local memory. This saves memory copying and reduces additional resource consumption.

[0083] Figure 9The diagram illustrates the execution flow of a hardware scheduler according to an embodiment of the present invention. Specifically, the hardware scheduler retrieves the fusion acceleration strategy and its target block combination from the block queue, parses the type of each block and the dependency graph determined by the operator relationship type, thus realizing the enqueueing and parsing of the fusion acceleration strategy and its target block combination. The dependency tracker monitors the completion signal of each block in real time. For each block that has not yet been executed, it checks whether all its dependencies are ready: if ready, the block is marked as ready and the dependency check result is sent to the resource allocator; if not ready, the block is suspended in the waiting queue, thus realizing dependency check judgment. Based on the dependency check result, the resource allocator allocates stream processors, registers or shared memory resources to ready computation blocks and allocates communication engines to ready communication blocks, thus realizing dynamic resource scheduling. Subsequently, the resource scheduler determines the launch order based on the communication execution type: under the guaranteed order type, ready blocks are launched sequentially in ascending order of block index; under the out-of-order type, all ready blocks can be launched in any order. When a communication block is transmitted, if the communication flow depends on the computation flow, the resource scheduler triggers a preset storage instruction; if the computation flow depends on the communication flow, it triggers a preset load instruction. Through unified memory addressing mapping, data from local registers is directly written to the remote computing card memory, or data is directly read from remote memory and written back to registers. During execution, the computing unit does not need to participate in data copying; the stream processor only needs to issue an instruction to switch to processing other blocks, allowing each block to perform computation or communication operations. Simultaneously, the dependency tracker updates the memory region occupancy status according to a preset synchronization mechanism: after the sending end executes the storage instruction, it automatically issues a release signal; the receiving end hardware marks the region as available via a bitmap and wakes up blocks in the waiting queue that depend on that region, writing back the results and notifying the dependent party. This cycle continues until all blocks have been executed, at which point dequeueing and resource reclamation operations are performed.

[0084] In this embodiment of the invention, the implementation process of the preset synchronization mechanism can be as follows: A bitmap and a waiting queue are maintained in the receiving end hardware. The bitmap is used to mark the current occupancy status of each memory region. When the sending end needs to access a memory region, it performs an acquisition operation: the hardware checks the flag of the region in the bitmap. If it is not occupied, it immediately marks it as occupied and allows access; if it is occupied, the current request is added to the waiting queue. After the sending end completes the access, it performs a release operation: the hardware clears the occupancy flag of the region in the bitmap, and then checks if there is an operation waiting for the region in the waiting queue. If so, the next request in the queue is woken up. In this way, the responsibility for ensuring order can be transferred from the sending end software to the receiving end hardware. The sending end only needs to perform acquisition or release operations in pairs without blocking, without explicit waiting or polling, thereby eliminating round-trip overhead, achieving zero-overhead synchronization, fully pipelined execution, and supporting fine-grained parallel access to different memory regions, i.e., maintaining order within the same region, while allowing complete parallelism between different regions.

[0085] The technical solution of this invention involves obtaining the target model architecture, target distributed parallel strategy, and target cluster architecture corresponding to the target fusion acceleration task. Based on the target model architecture and target distributed parallel strategy, the target communication computation operator combination and the operator relationship type corresponding to the target communication computation operator combination are determined. Semantic analysis of the target distributed parallel strategy is performed based on a preset communication library specification to determine the communication execution type corresponding to the target communication computation operator combination. The cluster architecture configuration corresponding to the target cluster architecture is determined based on the hardware configuration file, and the workload characteristics corresponding to the target communication computation operator combination are calculated based on the model hyperparameters corresponding to the target model architecture and the target distributed parallel strategy. Then, if the preset operator type is a computation operator type, the peak computing power and computational efficiency of a single card in the cluster architecture configuration are obtained. The computational quantity corresponding to the target computation operator in the target communication computation operator combination is calculated based on the workload characteristics, and the first execution time corresponding to the target computation operator is determined based on the peak computing power, computational efficiency, and computational quantity of a single card. If the preset operator type is a communication operator type, the cluster interconnection bandwidth and communication efficiency in the cluster architecture configuration are obtained. Based on workload characteristics, the data volume corresponding to the target communication operator in the target communication computing operator combination is calculated, and the second execution time corresponding to the target communication operator is determined based on the cluster interconnection bandwidth, communication efficiency, and data volume. The first and second execution times are combined as the base execution time corresponding to the target communication computing operator combination. Further, the target communication computing operator combination is sliced ​​and divided into blocks based on a preset candidate space to determine the candidate block set corresponding to the target communication computing operator combination. Execution time analysis is performed on each candidate block combination based on a preset search algorithm and the base execution time, and the target block combination corresponding to the target communication computing operator combination is determined based on the time analysis results. The fusion acceleration strategy corresponding to the target communication computing operator combination is determined based on the operator relationship type, communication execution type, and target block combination. Finally, the fusion acceleration strategy is parsed and processed through a block queue to obtain the communication computing dependencies corresponding to the fusion acceleration strategy. A dependency tracker performs dependency checks on the communication computing dependencies based on the target block combination and operator relationship type to obtain the dependency check results. The resource allocator determines the computing units and communication engines corresponding to the target block combination in the fusion acceleration strategy based on dependency check results. The resource scheduler controls the execution order and execution mode of the computing units and communication engines based on preset synchronization mechanisms and preset communication instructions to achieve the target fusion acceleration task.By taking a holistic approach from multiple levels, including system goal perception, adaptive optimization of segmentation granularity, hardware-level data path optimization, and low-overhead hardware synchronization mechanisms, communication and computation are decomposed into fine-grained blocks. The hardware maintains the dependencies between blocks, automatically handles data dependencies, and the synchronization signal mechanism between computation and communication blocks is completed by hardware. This solves the problems of low efficiency of computing fusion and insufficient utilization of computing card resources in the distributed training and inference of existing large language models. It can improve the hardware resource utilization of computing fusion, reduce the latency of computing fusion runtime, maximize the utilization of computation and memory access efficiency, and thus improve the model running efficiency.

[0086] Figure 10 The diagram shows a flowchart of an optional fusion acceleration method based on a distributed parallel strategy provided by an embodiment of the present invention. Specifically, firstly, the operator relationship type, communication execution type, cluster architecture configuration, and workload characteristics corresponding to the target communication computation operator combination in the target fusion acceleration task are obtained. Then, the basic execution time corresponding to the target communication computation operator combination is calculated based on the preset operator type, cluster architecture configuration, and workload characteristics. Further, the target communication computation operator combination is sliced ​​and divided into blocks based on a preset candidate space to determine the candidate block set corresponding to the target communication computation operator combination. The execution time of each candidate block combination is analyzed based on a preset search algorithm and the basic execution time, and the target block combination corresponding to the target communication computation operator combination is determined based on the time analysis results. Thus, the optimal block combination is obtained through automatic optimization. Furthermore, the kernel function is initiated, and the fusion acceleration strategy is parsed and processed through the block queue to obtain the communication computation dependencies corresponding to the fusion acceleration strategy. A dependency tracker performs dependency checks on the communication computation dependencies based on the target block combination and operator relationship type, obtaining the dependency check results. A resource allocator determines the computing units and communication engines corresponding to the target block combination in the fusion acceleration strategy based on the dependency check results. A resource scheduler controls the execution order and execution mode of the computing units and communication engines based on a preset synchronization mechanism and preset communication instructions. Thus, hardware scheduling manages the data dependencies between computing blocks and communication blocks. Preset communication instructions ensure that after the computing unit completes its computing block, the communication block is directly written from shared memory or registers to the communication network card buffer, and the network card writes to the remote computing card memory. A preset synchronization mechanism reduces the synchronization overhead of explicit waiting or polling. It also allows for parallel masking of computing blocks and communication blocks between different block objects. The kernel function completes its task, achieving the target fusion acceleration, until the last block object in the target block combination completes its computation or communication.

[0087] Example 3 Figure 11This is a schematic diagram of a fusion acceleration device based on a distributed parallel strategy provided in Embodiment 3 of the present invention. This device is applied to model training or inference processes using a distributed parallel strategy, such as... Figure 11 As shown, the device includes: a data acquisition module 310, a time calculation module 320, a strategy generation module 330, and a strategy execution module 340; Among them, the data acquisition module 310 is used to acquire the operator relationship type, communication execution type, cluster architecture configuration and workload characteristics corresponding to the target communication computing operator combination in the target fusion acceleration task; The time calculation module 320 is used to calculate the basic execution time corresponding to the target communication computing operator combination based on the preset operator type, the cluster architecture configuration, and the workload characteristics. The strategy generation module 330 is used to slice and divide the target communication computing operator combination based on the basic execution time to obtain the target block combination corresponding to the target communication computing operator combination, and to determine the fusion acceleration strategy corresponding to the target communication computing operator combination based on the operator relationship type, communication execution type and target block combination. The strategy execution module 340 is used to control the execution order and execution mode of the computing unit and the communication engine based on the fusion acceleration strategy through the hardware scheduler, so as to realize the target fusion acceleration task.

[0088] The technical solution of this invention obtains the operator relationship type, communication execution type, cluster architecture configuration, and workload characteristics corresponding to the target communication computing operator combination in the target fusion acceleration task. Then, based on the preset operator type, cluster architecture configuration, and workload characteristics, the basic execution time corresponding to the target communication computing operator combination is calculated. Further, based on the basic execution time, the target communication computing operator combination is sliced ​​and divided into blocks to obtain the target block combination corresponding to the target communication computing operator combination. Then, based on the operator relationship type, communication execution type, and target block combination, the fusion acceleration strategy corresponding to the target communication computing operator combination is determined. Finally, the hardware scheduler controls the execution order and execution mode of the computing unit and communication engine based on the fusion acceleration strategy to realize the target fusion acceleration task. By taking a holistic approach from multiple levels, including system goal perception, adaptive optimization of segmentation granularity, hardware-level data path optimization, and low-overhead hardware synchronization mechanisms, communication and computation are decomposed into fine-grained blocks. The hardware maintains the dependencies between blocks, automatically handles data dependencies, and the synchronization signal mechanism between computation and communication blocks is completed by hardware. This solves the problems of low efficiency of computing fusion and insufficient utilization of computing card resources in the distributed training and inference of existing large language models. It can improve the hardware resource utilization of computing fusion, reduce the latency of computing fusion runtime, maximize the utilization of computation and memory access efficiency, and thus improve the model running efficiency.

[0089] Optional, the time calculation module 320 can be used for: If the preset operator type is a computing operator type, then obtain the peak computing power and computing efficiency of a single card in the cluster architecture configuration, calculate the computing volume corresponding to the target computing operator in the target communication computing operator combination based on the workload characteristics, and determine the first execution time corresponding to the target computing operator based on the peak computing power, computing efficiency and computing volume of the single card. If the preset operator type is a communication operator type, then obtain the cluster interconnect bandwidth and communication efficiency in the cluster architecture configuration, calculate the data volume corresponding to the target communication operator in the target communication computing operator combination based on the workload characteristics, and determine the second execution time corresponding to the target communication operator based on the cluster interconnect bandwidth, communication efficiency and data volume. The first execution time and the second execution time are combined and processed as the base execution time corresponding to the target communication computing operator combination.

[0090] Optional, the time calculation module 320 can be used for: Based on the target computing operators in the target communication computing operator combination, the corresponding first feature parameter set is determined in the workload characteristics; Based on the product of each first feature parameter in the first feature parameter set, the computational amount corresponding to the target computation operator in the target communication computation operator combination is determined; Correspondingly, the time calculation module 320 can be used for: Based on the target communication operator in the target communication computation operator combination, the corresponding second feature parameter set is determined in the workload characteristics; Based on the product of each second feature parameter in the second feature parameter set, the data volume corresponding to the target communication operator in the target communication computation operator combination is determined.

[0091] Optionally, the data acquisition module 310 can be used for: Obtain the target model architecture, target distributed parallel strategy, and target cluster architecture corresponding to the target fusion acceleration task; Based on the target model architecture and the target distributed parallel strategy, the target communication computation operator combination corresponding to the target fusion acceleration task and the operator relationship type corresponding to the target communication computation operator combination are determined; wherein, the target communication computation operator combination includes at least one target computation operator and one target communication operator; Based on the preset communication library specifications, semantic analysis is performed on the target distributed parallel strategy to determine the communication execution type corresponding to the combination of target communication computation operators. The cluster architecture configuration corresponding to the target cluster architecture is determined based on the hardware configuration file, and the workload characteristics corresponding to the target communication computing operator combination are calculated based on the model hyperparameters corresponding to the target model architecture and the target distributed parallel strategy.

[0092] Optionally, the policy generation module 330 can be used for: The target communication computation operator combination is sliced ​​and divided into blocks based on a preset candidate space to determine the candidate block set corresponding to the target communication computation operator combination; wherein, the candidate block set contains at least one candidate block combination, and each candidate block combination contains at least one computation block and a corresponding communication block; The execution time of each candidate block combination is analyzed based on the preset search algorithm and the basic execution time, and the target block combination corresponding to the target communication computing operator combination is determined based on the time analysis results.

[0093] Optionally, the policy generation module 330 can be used for: Based on the operator relationship type and communication execution type corresponding to the target communication computation operator combination, the total overlap time model corresponding to the candidate block combination is determined, and the basic total overlap time corresponding to the candidate block combination is determined based on the total overlap time model. Execution time analysis is performed on the total time of each basic overlap based on the basic execution time to determine the target total overlap time that satisfies the basic execution time, and the candidate block combination corresponding to the target total overlap time is used as the target block combination corresponding to the target communication computing operator combination.

[0094] Optionally, the hardware scheduler includes a chunk queue, a dependency tracker, a resource allocator, and a resource scheduler; The strategy execution module 340 can be used for: The fusion acceleration strategy is parsed and processed by a block queue to obtain the communication computing dependencies corresponding to the fusion acceleration strategy. The dependency check results are obtained by performing dependency checks on the communication computation dependencies based on the target block combination and operator relationship type using a dependency tracker. The resource allocator determines the computing units and communication engines corresponding to the target block combination in the fusion acceleration strategy based on the dependency check results. The resource scheduler controls the execution order and method of the computing units and communication engine based on a preset synchronization mechanism and preset communication instructions to achieve target fusion acceleration tasks. The preset synchronization mechanism is a hardware-maintained bitmap and waiting queue synchronization mechanism. The preset communication instructions include storage instructions and loading instructions. The storage instructions are used in scenarios where the communication flow depends on the computing flow, directly writing data to the remote computing card memory through unified memory addressing mapping. The loading instructions are used in scenarios where the computing flow depends on the communication flow, directly reading data from the remote computing card memory through unified memory addressing mapping. When the computing unit executes the storage or loading instructions, it directly triggers network transmission, bypassing memory copying and reducing streaming processor resource consumption.

[0095] The fusion acceleration device based on the distributed parallel strategy provided in the embodiments of the present invention can execute the fusion acceleration method based on the distributed parallel strategy provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0096] Example 4 Figure 12 A schematic diagram of an electronic device 410 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0097] like Figure 12 As shown, the electronic device 410 includes at least one processor 420 and a memory, such as a read-only memory 430 or a random access memory 440, communicatively connected to the at least one processor 420. The memory stores computer programs executable by the at least one processor. The processor 420 can perform various appropriate actions and processes based on the computer program stored in the read-only memory 430 or loaded from storage unit 490 into the random access memory 440. The random access memory 440 may also store various programs and data required for the operation of the electronic device 410. The processor 420, read-only memory 430, and random access memory 440 are interconnected via a bus 450. An input / output interface 460 is also connected to the bus 450.

[0098] Multiple components in electronic device 410 are connected to input / output interface 460, including: input unit 470, such as keyboard, mouse, etc.; output unit 480, such as various types of monitors, speakers, etc.; storage unit 490, such as disk, optical disk, etc.; and communication unit 4100, such as network card, modem, wireless transceiver, etc. Communication unit 4100 allows electronic device 410 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0099] Processor 420 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 420 include, but are not limited to, central processing units, graphics processing units, various special-purpose artificial intelligence computing chips, various processors running machine learning model algorithms, digital signal processors, and any suitable processor, controller, microcontroller, etc. Processor 420 performs the various methods and processes described above, such as fusion acceleration methods based on distributed parallel strategies.

[0100] This method is applied to model training or inference processes using a distributed parallel strategy, and includes: Obtain the operator relationship type, communication execution type, cluster architecture configuration, and workload characteristics corresponding to the combination of target communication computation operators in the target fusion acceleration task; The base execution time corresponding to the target communication computing operator combination is calculated based on the preset operator type, the cluster architecture configuration, and the workload characteristics. The target communication computing operator combination is sliced ​​and divided into blocks based on the basic execution time to obtain the target block combination corresponding to the target communication computing operator combination. The fusion acceleration strategy corresponding to the target communication computing operator combination is determined based on the operator relationship type, communication execution type and target block combination. The hardware scheduler controls the execution order and execution mode of the computing unit and communication engine based on the fusion acceleration strategy to achieve the target fusion acceleration task.

[0101] In some embodiments, the fusion acceleration method based on a distributed parallel strategy can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 490. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 410 via read-only memory 430 and / or communication unit 4100. When the computer program is loaded into random access memory 440 and executed by processor 420, one or more steps of the fusion acceleration method based on a distributed parallel strategy described above can be performed. Alternatively, in other embodiments, processor 420 can be configured to execute the fusion acceleration method based on a distributed parallel strategy by any other suitable means (e.g., by means of firmware).

[0102] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays, application-specific integrated circuits (ASICs), application-specific standard products (ASICs), system-on-a-chip (SoCs), complex programmable logic devices, computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0103] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0104] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory, optical fibers, portable compact disk read-only memory, optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0105] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a cathode ray tube or liquid crystal display monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0106] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0107] A computing system can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product within the cloud computing service system to address the shortcomings of traditional physical hosts and virtual private servers, such as high management difficulty and weak business scalability.

[0108] This application also discloses a computer program product, which includes a computer program that, when executed by a processor, implements the fusion acceleration method based on a distributed parallel strategy provided in any embodiment of this application. This program product and the fusion acceleration method based on a distributed parallel strategy disclosed in the embodiments of this application belong to the same inventive concept, and therefore will not be described in detail here.

[0109] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0110] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A fusion acceleration method based on a distributed parallel strategy, characterized in that, The method is applied to model training or inference processes using a distributed parallel strategy, and the method includes: Obtain the operator relationship type, communication execution type, cluster architecture configuration, and workload characteristics corresponding to the combination of target communication computation operators in the target fusion acceleration task; The base execution time corresponding to the target communication computing operator combination is calculated based on the preset operator type, the cluster architecture configuration, and the workload characteristics. The target communication computing operator combination is sliced ​​and divided into blocks based on the basic execution time to obtain the target block combination corresponding to the target communication computing operator combination. The fusion acceleration strategy corresponding to the target communication computing operator combination is determined based on the operator relationship type, communication execution type and target block combination. The hardware scheduler controls the execution order and execution mode of the computing unit and communication engine based on the fusion acceleration strategy to achieve the target fusion acceleration task.

2. The method according to claim 1, characterized in that, The calculation of the basic execution time corresponding to the target communication computing operator combination based on the preset operator type, the cluster architecture configuration, and workload characteristics includes: If the preset operator type is a computing operator type, then obtain the peak computing power and computing efficiency of a single card in the cluster architecture configuration, calculate the computing volume corresponding to the target computing operator in the target communication computing operator combination based on the workload characteristics, and determine the first execution time corresponding to the target computing operator based on the peak computing power, computing efficiency and computing volume of the single card. If the preset operator type is a communication operator type, then obtain the cluster interconnect bandwidth and communication efficiency in the cluster architecture configuration, calculate the data volume corresponding to the target communication operator in the target communication computing operator combination based on the workload characteristics, and determine the second execution time corresponding to the target communication operator based on the cluster interconnect bandwidth, communication efficiency and data volume. The first execution time and the second execution time are combined and processed as the base execution time corresponding to the target communication computing operator combination.

3. The method according to claim 2, characterized in that, The calculation of the computational quantity corresponding to the target computational operator in the target communication computational operator combination based on workload characteristics includes: Based on the target computing operators in the target communication computing operator combination, the corresponding first feature parameter set is determined in the workload characteristics; Based on the product of each first feature parameter in the first feature parameter set, the computational amount corresponding to the target computation operator in the target communication computation operator combination is determined; The calculation of the data volume corresponding to the target communication operator in the target communication computing operator combination based on workload characteristics includes: Based on the target communication operator in the target communication computation operator combination, the corresponding second feature parameter set is determined in the workload characteristics; Based on the product of each second feature parameter in the second feature parameter set, the data volume corresponding to the target communication operator in the target communication computation operator combination is determined.

4. The method according to claim 1, characterized in that, The acquisition of the operator relationship type, communication execution type, cluster architecture configuration, and workload characteristics corresponding to the target communication computation operator combination in the target fusion acceleration task includes: Obtain the target model architecture, target distributed parallel strategy, and target cluster architecture corresponding to the target fusion acceleration task; Based on the target model architecture and the target distributed parallel strategy, the target communication computation operator combination corresponding to the target fusion acceleration task and the operator relationship type corresponding to the target communication computation operator combination are determined; wherein, the target communication computation operator combination includes at least one target computation operator and one target communication operator; Based on the preset communication library specifications, semantic analysis is performed on the target distributed parallel strategy to determine the communication execution type corresponding to the combination of target communication computation operators. The cluster architecture configuration corresponding to the target cluster architecture is determined based on the hardware configuration file, and the workload characteristics corresponding to the target communication computing operator combination are calculated based on the model hyperparameters corresponding to the target model architecture and the target distributed parallel strategy.

5. The method according to claim 1, characterized in that, The step of slicing and dividing the target communication computation operator combination based on the base execution time to obtain the target block combination corresponding to the target communication computation operator combination includes: The target communication computation operator combination is sliced ​​and divided into blocks based on a preset candidate space to determine the candidate block set corresponding to the target communication computation operator combination; wherein, the candidate block set contains at least one candidate block combination, and each candidate block combination contains at least one computation block and a corresponding communication block; The execution time of each candidate block combination is analyzed based on the preset search algorithm and the basic execution time, and the target block combination corresponding to the target communication computing operator combination is determined based on the time analysis results.

6. The method according to claim 5, characterized in that, The step of performing execution time analysis on each candidate block combination based on a preset search algorithm and basic execution time, and determining the target block combination corresponding to the target communication computation operator combination based on the time analysis results, includes: Based on the operator relationship type and communication execution type corresponding to the target communication computation operator combination, the total overlap time model corresponding to the candidate block combination is determined, and the basic total overlap time corresponding to the candidate block combination is determined based on the total overlap time model. Execution time analysis is performed on the total time of each basic overlap based on the basic execution time to determine the target total overlap time that satisfies the basic execution time, and the candidate block combination corresponding to the target total overlap time is used as the target block combination corresponding to the target communication computing operator combination.

7. The method according to claim 1, characterized in that, The hardware scheduler includes a block queue, a dependency tracker, a resource allocator, and a resource scheduler; The step of controlling the execution order and execution mode of the computing unit and communication engine based on the fusion acceleration strategy through a hardware scheduler to achieve the target fusion acceleration task includes: The fusion acceleration strategy is parsed and processed by a block queue to obtain the communication computing dependencies corresponding to the fusion acceleration strategy. The dependency check results are obtained by performing dependency checks on the communication computation dependencies based on the target block combination and operator relationship type using a dependency tracker. The resource allocator determines the computing units and communication engines corresponding to the target block combination in the fusion acceleration strategy based on the dependency check results. The resource scheduler controls the execution order and method of the computing units and communication engine based on a preset synchronization mechanism and preset communication instructions to achieve target fusion acceleration tasks. The preset synchronization mechanism is a hardware-maintained bitmap and waiting queue synchronization mechanism. The preset communication instructions include storage instructions and loading instructions. The storage instructions are used in scenarios where the communication flow depends on the computing flow, directly writing data to the remote computing card memory through unified memory addressing mapping. The loading instructions are used in scenarios where the computing flow depends on the communication flow, directly reading data from the remote computing card memory through unified memory addressing mapping. When the computing unit executes the storage or loading instructions, it directly triggers network transmission, bypassing memory copying and reducing streaming processor resource consumption.

8. A fusion acceleration device based on a distributed parallel strategy, characterized in that, The apparatus is used in model training or inference processes employing a distributed parallel strategy, and the apparatus includes: The data acquisition module is used to acquire the operator relationship type, communication execution type, cluster architecture configuration and workload characteristics corresponding to the combination of target communication computing operators in the target fusion acceleration task; The time calculation module is used to calculate the basic execution time corresponding to the target communication computing operator combination based on the preset operator type, the cluster architecture configuration, and the workload characteristics. The strategy generation module is used to slice and divide the target communication computing operator combination based on the basic execution time to obtain the target block combination corresponding to the target communication computing operator combination, and to determine the fusion acceleration strategy corresponding to the target communication computing operator combination based on the operator relationship type, communication execution type and target block combination. The strategy execution module is used to control the execution order and execution mode of the computing unit and the communication engine based on the fusion acceleration strategy through the hardware scheduler, so as to realize the target fusion acceleration task.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to execute the fusion acceleration method based on the distributed parallel strategy according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the fusion acceleration method based on a distributed parallel strategy as described in any one of claims 1-7.

Images