Computing and communication fusion operator compiling method and device based on many-core architecture

Abstract

The embodiment of the invention discloses a calculation and communication fusion operator compiling method and device based on a many-core architecture, which are applied to large-model multi-card heterogeneous compiling deployment, and the method comprises the following steps: fusing a communication operator and a calculation operator on a calculation graph into a fusion operator based on a preset operator fusion rule; calling a pre-constructed fusion cost model to evaluate the fusion operator, performing multi-core grouping optimization according to an evaluation result, and dividing multiple cores into a calculation group and a communication group; judging whether a preset performance target is met or not; if yes, generating an execution plan and deploying execution; the method has the beneficial effects that a segmentation and scheduling scheme fusing operator fine grit can be established, calculation and communication are maximally parallel, the operation time is optimized, and the utilization rate of a calculation / communication unit is effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of large model reasoning technology, and in particular to a method and device for compiling computing and communication fusion operators based on a many-core architecture. Background Art

[0002] Large language models (LLMs) are developing rapidly and are increasingly being applied to various scenarios requiring language processing, such as machine translation, intelligent query, and code debugging. However, because large language models contain hundreds of billions of parameters, far exceeding the storage capacity of a single processor, a single processor cannot fully perform the entire inference process. This has led to the use of multi-core processor architectures to support inference services for large models, which in turn involves computational communication between different processors.

[0003] Existing technologies typically treat computation and inter-card communication as two independent tasks, employing serial or coarse-grained (multi-batch) computation and communication parallelism. Especially on multi-core processors, computation libraries are typically deployed to optimize inter-core computation and communication, while communication libraries are deployed to optimize inter-card communication and computation. However, support for parallel optimization between cores and cards is limited. Due to uneven computational load, communication latency, and synchronization issues, achieving reasonable computational resource and bandwidth utilization in multi-card parallel inference scenarios is difficult. Summary of the Invention

[0004] In view of the technical deficiencies of the prior art, the embodiment of the present invention aims to provide a method and apparatus for compiling computing and communication fusion operators based on a many-core architecture.

[0005] To achieve the above objectives, in a first aspect, an embodiment of the present invention provides a method for compiling a computing and communication fusion operator based on a many-core architecture, which is applied to large-model multi-card heterogeneous compilation deployment. The method includes:

[0006] Based on the preset operator fusion rules, the communication operator and the calculation operator on the calculation graph are fused into a fusion operator;

[0007] Calling a pre-built fusion cost model to evaluate the fusion operator, and performing multi-core grouping optimization based on the evaluation results, dividing the multi-cores into computing groups and communication groups;

[0008] Determine whether the preset performance goals are met; if so, generate an execution plan and deploy it for execution.

[0009] As a specific implementation of the present application, the operator fusion rule follows the combination of two consecutive computing operators and a communication operator.

[0010] As a specific implementation of this application, the evaluation specifically includes:

[0011] Number of perception cards;

[0012] The computational and communication overheads in fusion operators are evaluated to provide guidance for multi-core grouping.

[0013] As a specific implementation of this application, task-level parallelism is adopted within the group. The computing group processes the operator computing tasks, and the communication group is responsible for the data collection and interactive processing required for inter-card communication;

[0014] Pipeline parallelism is adopted between groups. Based on the data segmentation within the fusion operator, the calculation group outputs data and collects it to the communication group for subsequent inter-card transmission and processing. At the same time, the calculation group processes the next set of data.

[0015] As a specific implementation method of the present application, based on the relevant information of a given fusion operator, the proportion of computing overhead and communication overhead is analyzed to determine the multi-core grouping strategy; wherein, the ratio of computing overhead and communication overhead should be close to the weighted ratio of the number of cores in the computing group to the number of cores in the communication group, and the operator calculation and communication time should be close to each other.

[0016] As a specific implementation of the present application, the many-core processors in the many-core architecture support grouped instruction issuance, synchronization between multiple groups of processors, and support parallel memory access by computing components and communication components.

[0017] In a second aspect, an embodiment of the present invention further provides a computing and communication fusion operator compilation device based on a many-core architecture, which is applied to large-model multi-card heterogeneous compilation deployment, and the device includes:

[0018] A fusion module is used to fuse the communication operator and the computation operator on the computation graph into a fusion operator based on a preset operator fusion rule;

[0019] A processing module is used to call a pre-built fusion cost model to evaluate the fusion operator, and perform multi-core grouping optimization based on the evaluation result, dividing the multi-core into a computing group and a communication group;

[0020] The judgment module is used to determine whether the preset performance goals are met; if so, an execution plan is generated and deployed for execution.

[0021] The technical solution provided by the embodiment of the present invention realizes the pipelining of computing and communication through operator fusion, analyzes the computing and communication overhead of the fusion operator, and divides multiple cores into two groups, one group is responsible for intra-card computing, and the other group is responsible for inter-card communication and computing; thereby establishing a fine-grained segmentation and scheduling scheme for the fusion operator, achieving maximum parallelism of computing and communication, optimizing the running time, and effectively improving the utilization rate of computing / communication units. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the specific implementation of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the specific implementation or the description of the prior art.

[0023] Figure 1 This is a flowchart of a method for compiling a computing and communication fusion operator based on a many-core architecture provided by an embodiment of the present invention;

[0024] Figure 2 This is a schematic diagram of a deployment process provided by an embodiment of the present invention;

[0025] Figure 3 This is an example diagram of multi-core grouping provided by an embodiment of the present invention;

[0026] Figure 4 This is a structural block diagram of a computing and communication fusion operator compilation device based on a many-core architecture provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0027] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0028] It will be understood that when used in this specification and the appended claims, the terms “comprises” and “comprising” indicate the presence of described features, integers, steps, operations, elements and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0029] Throughout this specification, references to "one embodiment," "an embodiment," "an example," or "an example" mean that a particular feature, structure, or characteristic described in connection with the embodiment or example is included in at least one embodiment of the present invention. Thus, the appearances of the phrases "in one embodiment," "in an embodiment," "an example," or "an example" in various places throughout this specification are not necessarily all referring to the same embodiment or example. Furthermore, the particular features, structures, or characteristics may be combined in any suitable combinations and / or subcombinations in one or more embodiments or examples.

[0030] Tensor: In the field of artificial intelligence, a tensor is a multidimensional array.

[0031] Batch: refers to the set of samples processed simultaneously in one iteration; during the training or inference process, not all data samples are processed at once, but the data is divided into several small sets, each of which is a "batch".

[0032] Inter-Card Communication: refers to the data transmission and collaborative working mechanism between different computing cards (such as GPUs, AI accelerator cards, etc.), usually achieved through high-speed interconnect technologies (such as NVLink and InfiniBand) for distributed computing or large-scale parallel tasks.

[0033] Many-core architecture: A processor architecture that integrates a large number of computing cores (usually dozens to thousands), improves throughput through parallel computing, and is suitable for scenarios such as high-performance computing (HPC) and AI training.

[0034] Operator: The basic computing unit in the deep learning framework, representing mathematical operations (such as matrix multiplication and pooling) or logical control (such as conditional judgment), is the basic component for building neural networks.

[0035] Fused Operator (or Operator Fusion): In deep learning, a technique that combines multiple basic operators (such as convolution and activation functions) into a single composite operation can reduce memory access overhead and optimize computational efficiency. For example, "Conv+ReLU" can be fused into a single operator.

[0036] FLOPs (Floating Point Operations): A measure of the computational complexity of an algorithm or model, representing the total number of floating-point operations (such as addition and multiplication) required during execution. It is an important metric for evaluating the efficiency of computing tasks.

[0037] It should be noted that, unless otherwise specified, the technical terms in this embodiment have the common meanings understood in the relevant technical field.

[0038] Please refer to Figures 1 to 3 The embodiment of the present invention provides a method for compiling a computing and communication fusion operator based on a many-core architecture, which is applied to large-model multi-card heterogeneous compilation deployment. The method includes:

[0039] S101, fusing the communication operator and the computation operator on the computation graph into a fusion operator based on a preset operator fusion rule;

[0040] S102, calling a pre-built fusion cost model to evaluate the fusion operator, and performing multi-core grouping optimization based on the evaluation result, dividing the multi-core into a computing group and a communication group;

[0041] S103, determining whether the preset performance target is met; if so, generating an execution plan and deploying it for execution.

[0042] During implementation, the many-core processors in the many-core architecture support grouped instruction delivery, synchronization between multiple groups of processors, and parallel memory access by computing components and communication components;

[0043] The performance targets include bandwidth (communication transmission time) and computing power; computing power can be understood as computing time;

[0044] The execution plan refers to the compilation process, which converts the computing and communication fusion operators into an instruction file that can be executed on the current many-core processor.

[0045] The operator fusion rule follows the combination of two consecutive computing operators plus a communication operator;

[0046] In existing LLM multi-card parallelization methods, there are usually methods such as Gemm + multi-card all reduce fusion and Gemm + multi-card allgather fusion. This allows the fusion operator to represent both the operator calculation logic and the connection between the multi-card model calculation graphs. Among them, the connection can be understood as the synchronous execution of the current calculation graph on different cards.

[0047] In this embodiment, the evaluation specifically includes:

[0048] Number of perception cards;

[0049] Evaluate the computational and communication overheads of fusion operators to guide multi-core grouping;

[0050] At the same time, task-level parallelism is adopted within the group. The computing group processes operator computing tasks, and the communication group is responsible for data collection and interactive processing required for inter-card communication. The interactive processing includes data splicing and mathematical logic operations between data.

[0051] Pipeline parallelism is adopted between groups. Based on the data segmentation within the fusion operator, the calculation group outputs data and collects it to the communication group for subsequent inter-card transmission and processing. At the same time, the calculation group processes the next set of data.

[0052] The computational overhead includes FLOPs / latency / power consumption; the communication overhead includes bandwidth / latency / synchronization cost;

[0053] The computing group is responsible for on-card computing, and the communication group is responsible for inter-card communication and computing.

[0054] The construction of the fusion cost model includes:

[0055] Based on the relevant information of a given fusion operator, the proportion of computational overhead and communication overhead is analyzed to determine a multi-core grouping strategy. The computational overhead and communication overhead ratio should be close to the weighted ratio of the number of cores in the computation group to the number of cores in the communication group, and the operator computation and communication times should be aligned as much as possible (equivalent to being close to each other). The closeness can be understood as the optimal solution (multi-core grouping strategy) for the grouping relationship of the limited number of cores in the many-core processor. That is, the weighted ratio of the number of cores in the computation group to the communication group is closest to the computational and communication overhead ratios of the given fusion operator.

[0056] When applying it, we first model the backend hardware and then combine it with the task volume to evaluate the computational and communication overheads in the fusion operator and guide the grouping of multiple cores to achieve load balancing.

[0057] It's important to note that the fusion operator in this method differs from conventional operators primarily in that deployment and compilation require awareness of the number of cards involved, with the cores on each card divided into two groups. Therefore, a top-down three-layer abstraction hierarchy (card-group-core) is established to support scheduling and instruction generation. This breaks the limitations of traditional single-card compilation and achieves innovative compilation abstraction.

[0058] The above solution achieves pipelined computing and communication through operator fusion, analyzes the computational and communication overhead of the fusion operator, and divides multiple cores into two groups, one responsible for intra-card computing and the other for inter-card communication and computing. This establishes a fine-grained segmentation and scheduling scheme for the fusion operator, achieves maximum parallelism of computing and communication, optimizes runtime, and effectively improves the utilization of computing / communication units.

[0059] Based on the same inventive concept, the embodiment of the present invention also provides a computing and communication fusion operator compilation device based on a multi-core architecture, referring to Figure 4 , applied to large-model multi-card heterogeneous compilation deployment, the device includes:

[0060] A fusion module is used to fuse the communication operator and the computation operator on the computation graph into a fusion operator based on a preset operator fusion rule;

[0061] A processing module is used to call a pre-built fusion cost model to evaluate the fusion operator, and perform multi-core grouping optimization based on the evaluation result, dividing the multi-core into a computing group and a communication group;

[0062] The judgment module is used to determine whether the preset performance goals are met; if so, an execution plan is generated and deployed for execution.

[0063] Furthermore, the operator fusion rule follows the combination of two consecutive computing operators plus a communication operator; through operator fusion, the pipelining of computing and communication is achieved, and the unified scheduling of heterogeneous resources is realized.

[0064] The assessment specifically includes:

[0065] Number of perception cards;

[0066] Evaluate the computational and communication overheads of fusion operators to guide multi-core grouping;

[0067] At the same time, task-level parallelism is adopted within the group. The computing group processes operator computing tasks, and the communication group is responsible for data collection and interactive processing required for inter-card communication.

[0068] Pipeline parallelism is adopted between groups. Based on the data segmentation within the fusion operator, the calculation group outputs data and collects it to the communication group for subsequent inter-card transmission and processing. At the same time, the calculation group processes the next set of data.

[0069] It should be noted that for a more specific description of the workflow of the device embodiment, please refer to the aforementioned method embodiment part, which will not be repeated here.

[0070] Beneficial effects can be achieved: Taking a single inference scenario as an example, the latency of computing and inter-card communication can be hidden, reducing response time;

[0071] Improve computing / communication unit utilization; optimize storage, and reduce intermediate storage through operator fusion;

[0072] Increase compilation versatility.

[0073] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and such modifications or substitutions are intended to be within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A method for compiling operators for computing and communication fusion based on many-core architecture, characterized in that: Applied to large-model multi-GPU heterogeneous compilation deployment, the method includes: Based on the preset operator fusion rules, the communication operator and the calculation operator on the calculation graph are fused into a fusion operator; Calling a pre-built fusion cost model to evaluate the fusion operator, and performing multi-core grouping optimization based on the evaluation results, dividing the multi-cores into computing groups and communication groups; Determine whether the preset performance goals are met; if so, generate an execution plan and deploy it for execution.

2. The method according to claim 1, wherein The operator fusion rule follows the combination of two consecutive computing operators plus a communication operator.

3. The method according to claim 1 or 2, wherein: The assessment specifically includes: Number of perception cards; The computational and communication overheads in fusion operators are evaluated to provide guidance for multi-core grouping.

4. The method according to claim 3, wherein Task-level parallelism is adopted within the group. The computing group processes operator computing tasks, and the communication group is responsible for data collection and interaction processing required for inter-card communication. Pipeline parallelism is adopted between groups. Based on the data segmentation within the fusion operator, the calculation group outputs data and collects it to the communication group for subsequent inter-card transmission and processing. At the same time, the calculation group processes the next set of data.

5. The method according to claim 1, wherein The construction of the fusion cost model includes: Based on the relevant information of a given fusion operator, the proportion of computing overhead and communication overhead is analyzed to determine the multi-core grouping strategy; wherein, the ratio of computing overhead and communication overhead should be close to the weighted ratio of the number of cores in the computing group to the number of cores in the communication group, and the operator calculation and communication time should be close to each other.

6. The method according to claim 1, wherein The many-core processors in the many-core architecture support grouped instruction issuance, synchronization between multiple groups of processors, and parallel memory access by computing components and communication components.

7. A computing and communication fusion operator compilation device based on a many-core architecture, characterized in that: Applicable to large-scale model multi-card heterogeneous compilation deployment, the device includes: A fusion module is used to fuse the communication operator and the computation operator on the computation graph into a fusion operator based on a preset operator fusion rule; A processing module is used to call a pre-built fusion cost model to evaluate the fusion operator, and perform multi-core grouping optimization based on the evaluation result, dividing the multi-core into a computing group and a communication group; The judgment module is used to determine whether the preset performance goals are met; if so, an execution plan is generated and deployed for execution.

8. The computing and communication fusion operator compilation device based on many-core architecture according to claim 7, characterized in that: The assessment specifically includes: Number of perception cards; The computational and communication overheads in fusion operators are evaluated to provide guidance for multi-core grouping.

9. The computing and communication fusion operator compilation device based on many-core architecture according to claim 7, characterized in that: Task-level parallelism is adopted within the group. The computing group processes operator computing tasks, and the communication group is responsible for data collection and interaction processing required for inter-card communication. Pipeline parallelism is adopted between groups. Based on the data segmentation within the fusion operator, the calculation group outputs data and collects it to the communication group for subsequent inter-card transmission and processing. At the same time, the calculation group processes the next set of data.

Images