A communication and computation parallel method, device, apparatus and medium

Abstract

The application discloses a communication and calculation parallel method, device, equipment and medium. The method comprises the following steps: dividing input data into two data streams through a host end, and determining the function module sorting, grouping dependency relationship and execution dependency relationship of a single data stream according to a Transformer layer architecture; alternately sorting the function modules corresponding to the two data streams according to the function module sorting and the grouping dependency relationship of the single data stream through the host end, obtaining the issuing sequence of the function modules, and sequentially issuing each function module to the corresponding calculation stream or communication stream; and monitoring the function modules to be started in the calculation stream and the communication stream through an AI acceleration chip, and starting to execute the function module when it is monitored that the function module to be started meets the starting condition. By adopting the technical scheme, communication and calculation parallelism can be realized by using two data streams, and the overall throughput and resource utilization of the cluster are effectively improved.

Description

Technical Field

[0001] This invention relates to the field of large-scale artificial intelligence models, and in particular to a method, apparatus, device, and medium for parallel communication and computing. Background Technology

[0002] Large AI (Artificial Intelligence) models are typically composed of multiple stacked Transformer layers, each containing multiple communication and computation operations, resulting in a complex overall computation and communication process.

[0003] Existing solutions typically place communication operators and computation operators in the same hardware resource or the same execution flow, and use a single data flow to execute communication and computation operations within the Transformer layer sequentially. This results in communication and computation processes waiting for each other and being unable to overlap.

[0004] With the rapid development of AI large model technology, the model parameters and deployment cluster scale of AI large models are constantly increasing. The communication volume brought by the hybrid expert sparse computing (MoE) module in the Transformer layer is increasing dramatically. The serial execution mode of communication and computation will cause computing resources to be idle and wait. The proportion of communication time in the overall inference time continues to rise, becoming a performance bottleneck. Even if communication and computation operations are executed in parallel, the sharing of hardware resources between communication and computation will also affect computing performance, resulting in low overall cluster throughput and high latency, making it difficult to meet the requirements of large model inference scenarios with high real-time requirements. Summary of the Invention

[0005] This invention provides a method, apparatus, device, and medium for parallel communication and computing, which can achieve parallel communication and computing using two data streams, effectively improving the overall throughput and resource utilization of the cluster.

[0006] According to one aspect of the present invention, a parallel communication and computing method is provided, which is executed by a host computer in cooperation with an AI acceleration chip, comprising: On the host side, the input data of the large AI model is divided into two data streams, and the functional modules, grouping dependencies, and execution dependencies of each data stream are determined according to the Transformer layer architecture of the large AI model. The functional modules include computational modules and communication modules. On the host side, based on the functional module sorting and grouping dependency of the single data stream, the functional modules corresponding to the two data streams are alternately sorted to obtain the delivery order of the functional modules. Then, according to the delivery order of the functional modules, each functional module is sequentially delivered to the corresponding computing stream or communication stream. The AI ​​acceleration chip monitors the functional modules to be started in the computing and communication streams based on their execution status and the execution dependencies between them. When a functional module to be started meets the startup conditions, it is started and executed. The computing stream runs in the general computing unit of the AI ​​acceleration chip, and the communication stream runs in the communication computing unit of the AI ​​acceleration chip.

[0007] According to another aspect of the present invention, a parallel communication and computing device is provided, which is executed by a host computer and an AI acceleration chip, comprising: The sorting determination module is used to divide the input data of the large AI model into two data streams on the host side, and determine the sorting, grouping dependencies, and execution dependencies of the functional modules of each data stream according to the Transformer layer architecture of the large AI model; among them, the functional modules include computation modules and communication modules; The function module distribution module is used by the host to alternately sort the function modules corresponding to the two data streams according to the function module order and grouping dependency of the single data stream, obtain the distribution order of the function modules, and distribute each function module to the corresponding computing stream or communication stream in sequence according to the distribution order of the function modules. The function module startup module is used to monitor the function modules to be started in the computing flow and communication flow through the AI ​​acceleration chip, based on the execution status of the function modules in the computing flow and communication flow and the execution dependencies between the function modules. When the function module to be started meets the startup conditions, the module is started and executed. The computing flow runs in the general computing unit of the AI ​​acceleration chip, and the communication flow runs in the communication computing unit of the AI ​​acceleration chip.

[0008] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the communication and computation parallel method according to any embodiment of the present invention.

[0009] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the communication and computation parallel method described in any embodiment of the present invention.

[0010] The technical solution of this invention involves dividing the input data of a large AI model into two data streams on the host side, and determining the functional module order, grouping dependencies, and execution dependencies of each data stream based on the Transformer layer architecture of the large AI model. On the host side, based on the functional module order and grouping dependencies of each data stream, the functional modules corresponding to the two data streams are alternately ordered to obtain the delivery order of the functional modules, and each functional module is sequentially delivered to its corresponding computation stream or communication stream according to the delivery order. Finally, an AI acceleration chip is used to process the computation stream and communication stream... The system monitors the execution status of functional modules and the execution dependencies between them. It monitors the functional modules to be started in the computing and communication streams, and starts the execution of a functional module when the startup conditions are met. The computing stream runs in the general computing unit of the AI ​​acceleration chip, and the communication stream runs in the communication computing unit of the AI ​​acceleration chip. This isolation of computing and communication hardware resources ensures that communication and computing do not affect each other's performance. By alternately sending dual data streams, deep overlap between communication and computing can be achieved, hiding communication latency and effectively improving the overall throughput and resource utilization of the cluster.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a flowchart of a parallel communication and computing method provided according to Embodiment 1 of the present invention; Figure 2 This is an architecture diagram of a Transformer layer provided according to an embodiment of the present invention; Figure 3 This is a timing diagram of the execution of functional modules in a computation flow and a communication flow according to an embodiment of the present invention; Figure 4 This is a flowchart of another parallel communication and computing method provided according to Embodiment 2 of the present invention; Figure 5 This is a timing diagram of a multi-level pipelined asynchronous data transmission according to an embodiment of the present invention; Figure 6This is a flowchart of a multi-level pipelined asynchronous data transmission communication method according to Embodiment 3 of the present invention; Figure 7 This is a schematic diagram of the structure of a parallel communication and computing device according to Embodiment 4 of the present invention; Figure 8 This is a schematic diagram of the structure of an electronic device that implements the parallel communication and computing method of the present invention. Detailed Implementation

[0014] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0015] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0016] Example 1 Figure 1 This is a flowchart of a communication and computation parallel method provided in Embodiment 1 of the present invention. This embodiment is applicable to the situation of communication and computation parallelism in a large artificial intelligence model through two data streams. The method can be executed by a communication and computation parallel device, which can be implemented in hardware and / or software and is generally configured in a computer or processor with data processing capabilities. The computer or processor includes a host and an AI acceleration chip.

[0017] Optionally, the host can be the CPU (Central Processing Unit) in a computer. The host only performs scheduling and does not perform calculation or communication operations.

[0018] Optionally, AI acceleration chips are the core hardware in a computer that executes large AI models. AI acceleration chips can be GPUs (Graphics Processing Units). A computer can be configured with multiple AI acceleration chips. For example, a single machine with 8 cards means that a single computer is configured with 8 AI acceleration chips.

[0019] like Figure 1 As shown, the method includes: S110. On the host side, the input data of the AI ​​large model is divided into two data streams, and the functional module ordering, grouping dependency relationship and execution dependency relationship of the single data stream are determined according to the Transformer layer architecture of the AI ​​large model.

[0020] The functional modules include computing modules and communication modules.

[0021] Optionally, large AI models need to be hosted and run by a cluster, which can refer to a unified hardware system formed by multiple computers and multiple AI acceleration chips connected through a high-speed network.

[0022] Optionally, the data dimension of the input data for the AI ​​large model can be [batchsize, token_num, hidden_size], where batchsize represents the batch size of the input data, token_num represents the total number of tokens in the input data, and hidden_size represents the length of the feature vector corresponding to each token.

[0023] Optionally, the batch size of the input data can be understood as the number of samples input to the model, and the total number of lexical units can be understood as the total number of lexical units contained in all samples within the current batch.

[0024] Optionally, a word unit is the smallest data unit processed by the large AI model. It is the basic unit obtained after the sample is segmented. All calculations, transmissions and feature extractions of the model are performed on word units.

[0025] Optionally, the input data can be split along the batch size dimension via the host side, dividing the input data into two parts. After being fed into the AI ​​large model, it can form two data streams. There is no data dependency between the two data streams. The splitting rule can be average splitting. For example, if the batch size is 8, the first data stream is formed based on the first 4 samples, and the second data stream is formed based on the last 4 samples.

[0026] Optionally, Transformer is a neural network architecture based on a self-attention mechanism, which is the basic framework of large AI models.

[0027] Optional, Figure 2Here is an architecture diagram for an optional Transformer layer, such as... Figure 2 As shown, a single Transformer layer includes multiple functional modules, which may include a pre-attention calculation module, a core attention calculation module, a distribution communication module, a shared expert general calculation module, a hybrid expert sparse calculation module, and a merging communication module.

[0028] Optionally, the large AI model can be composed of multiple stacked Transformer layers, processing the term data layer by layer. The input and output data dimensions are the same between each layer, and each Transformer layer has the same characteristics. Figure 2 The structure shown is different for the same functional modules in different Transformer layers, but the weights of these modules are different.

[0029] Optionally, in the Transformer layer, the computation modules include a pre-attention computation module, a core attention computation module, a shared expert general computation module, and a hybrid expert sparse computation module, while the communication modules include a distribution communication module and a merging communication module.

[0030] Optionally, the computation module can be used to perform related calculations such as feature extraction and model inference, while the communication module can be used to realize the transmission and aggregation of word units between different AI acceleration chips.

[0031] like Figure 2 As shown, the input data of a single Transformer layer is sent to the pre-attention calculation module and processed according to the data flow of each module within the Transformer layer. Finally, the data output by the shared expert general calculation module and the merged communication module are fused and processed to output the output data of a single Transformer layer.

[0032] Optionally, for the first Transformer layer of the AI ​​large model, its input is the two data streams obtained by dividing the AI ​​large model. The two data streams are passed into the AI ​​large model for calculation in sequence. For the non-first Transformer layers of the AI ​​large model, its input is the output data of the previous Transformer layer. The data output by the last Transformer layer for the two data streams is integrated and can be used as the output data of the entire AI large model.

[0033] Optionally, the functional module sorting of a single data stream can refer to a fixed sorting of an independent data stream in the large AI model, from the first Transformer layer to the last Transformer layer, according to the forward computation logic of the model.

[0034] Optionally, within a single Transformer layer, the functional modules are ordered from first to last as follows: pre-attention calculation module, core attention calculation module, distribution communication module, shared expert general calculation module, hybrid expert sparse calculation module, and merged communication module.

[0035] Furthermore, by superimposing the order of the functional modules in each Transformer layer according to the sequence of the multiple Transformer layers in the AI ​​large model, the order of the functional modules in a single data stream can be determined.

[0036] Optionally, grouping dependency can refer to the constraint relationship of grouping functional modules, where modules within a group must be executed continuously and modules between groups can be scheduled independently, in order to achieve the alternating parallelism of two data streams.

[0037] Optionally, for a single Transformer layer, the grouping dependency relationship can be as follows: the pre-attention calculation module is grouped independently, the core attention calculation module, the distribution communication module, and the shared expert general calculation module are merged to form a continuous group, and the hybrid expert sparse calculation module and the merged communication module are merged to form a continuous group.

[0038] Optionally, execution dependency can refer to the constraint relationship between functional modules that the output of the preceding module must be used as the input of the following module. Only after the preceding module has completed execution can the following module start execution.

[0039] Optionally, for a single Transformer layer, execution dependencies may include: the pre-attention computation module is a prerequisite module for the core attention computation module; the core attention computation module can only be started after the pre-attention computation module has been executed; the core attention computation module is a prerequisite module for the distribution communication module and the shared expert general computation module; the distribution communication module can only be started after the core attention computation module has been executed; the distribution communication module is a prerequisite module for starting the hybrid expert sparse computation module; the hybrid expert sparse computation module can only be started after the distribution communication module has been executed; and the hybrid expert sparse computation module is a prerequisite module for the merged communication module; the merged communication module can only be started after the hybrid expert sparse computation module has been executed.

[0040] Optionally, between two Transformer layers, the pre-attention calculation module of the Transformer layer can only start execution after obtaining the output result of the previous Transformer layer.

[0041] Optionally, within a single Transformer layer, the functional modules, grouping dependencies, and execution dependencies within that layer are followed; between multiple Transformer layers, the next layer can only be executed after the previous layer has finished executing.

[0042] Optionally, the functional modules of each single Transformer layer can be sorted within the layer, grouped, and executed. Then, the functional modules can be stacked according to the order relationship between the multiple Transformer layers to determine the functional module sorting, grouping, and execution dependencies of a single data stream.

[0043] Optionally, there is no data dependency between the two data streams. Therefore, a single data stream only needs to consider its own grouping dependency and execution dependency.

[0044] S120. On the host side, based on the functional module sorting and grouping dependency of the single data stream, the functional modules corresponding to the two data streams are alternately sorted to obtain the distribution order of the functional modules, and each functional module is distributed to the corresponding computing stream or communication stream in sequence according to the distribution order of the functional modules.

[0045] Optionally, each data stream has a fixed functional module order. Through the host, multiple functional module groups can be divided in the functional module order of a single data stream according to the grouping dependency relationship. Then, the functional modules of the two data streams are interspersed to form a sequence of functional modules that integrates the two data streams, which is also the order in which the functional modules are distributed.

[0046] Optionally, the host can sequentially distribute each functional module to the corresponding execution flow in the AI ​​acceleration chip according to the order in which the functional modules are distributed. Computational modules are distributed to the computational flow, and communication modules are distributed to the communication flow, thereby achieving parallel communication and computation.

[0047] Optionally, the computational flow can refer to an execution flow specifically used to perform AI large model computation tasks, and the communication flow can refer to an execution flow specifically used for data transmission. In the embodiments of the present invention, the computational flow and the communication flow are isolated in hardware resources and executed asynchronously in parallel, and can run simultaneously, thereby achieving non-blocking and overlapping execution of computation and communication.

[0048] Specifically, based on the functional module sorting and grouping dependencies of a single data stream, the functional modules corresponding to the two data streams are alternately sorted to obtain the delivery order of the functional modules. This may include: On the host side, based on the functional module sorting and grouping dependency of the single data stream, the functional modules are grouped in the functional module sorting corresponding to the two data streams respectively; On the host side, the functional modules corresponding to the two data streams are sorted alternately by group to obtain the distribution order of the functional modules; among them, the first two groups of functional modules belong to the same data stream in the distribution order of the functional modules. The grouping dependencies include: the pre-attention calculation module is grouped separately; the core attention calculation module, the distribution communication module, and the shared expert general calculation module are grouped together; and the hybrid expert sparse calculation module and the merged communication module are grouped together.

[0049] Optionally, when alternating the sorting of modules in the two data streams, the first two groups of functional modules are taken from the same data stream, enabling the data stream to start quickly and enter a stable execution state; the subsequent groups of functional modules are alternately sent between the two data streams, so that the calculation and communication operations of the two data streams are interleaved and overlapped without blocking.

[0050] The advantage of this setup is that, through the above-mentioned alternating delivery method, the hardware isolation characteristics of the general computing unit and the communication computing unit can be fully utilized to achieve deep parallelism in computing and communication.

[0051] Figure 3 This is a timing diagram for the execution of functional modules in an optional computational and communication flow. For example... Figure 3 As shown, the modules of the two data streams are represented by white and gray backgrounds, respectively. The numbers inside the modules are used to indicate the layer number of the Transformer layer to which the module belongs. For example, the hybrid expert sparse computation module 1 with a gray background can represent the hybrid expert sparse computation module of the second data stream in the first Transformer layer, and the pre-attention computation module 2 with a white background can represent the pre-attention computation module of the first data stream in the second Transformer layer.

[0052] Optional, Figure 3 The diagram fully illustrates the timing relationship of the execution of each functional module within the first Transformer layer, as well as the connection relationship between the first and second Transformer layers. A single large AI model consists of multiple Transformer layers, and the execution timing relationship of the functional modules of all Transformer layers within the large AI model can be further arranged according to the inter-layer relationship.

[0053] Depend on Figure 3It can be seen that the host first sends two sets of functional modules from the first data stream to the corresponding computation or communication stream. The two sets of functional modules are an independent group containing the pre-attention computation module of the first Transformer layer, and a continuous group consisting of the core attention computation module, the distribution communication module, and the shared expert general computation module of the first Transformer layer. Then, a set of functional modules from the second data stream is sent to the corresponding computation or communication stream, which is the independent group containing the pre-attention computation module of the first Transformer layer. Then, a set of functional modules from the first data stream is sent again, and so on.

[0054] It can be further seen that after all the functional modules in the first Transformer layer of the first data stream have been deployed, the functional modules in the first Transformer layer of the second data stream have not been deployed yet. At this time, the functional modules in the second Transformer layer of the first data stream and the functional modules in the first Transformer layer of the second data stream that have not been deployed are deployed alternately according to module grouping, and so on.

[0055] S130. Using the AI ​​acceleration chip, based on the execution status of functional modules in the computing flow and communication flow and the execution dependencies between functional modules, monitor the functional modules to be started in the computing flow and communication flow, and start the execution of the functional module when the functional module to be started meets the start conditions.

[0056] The computing stream runs in the general computing unit of the AI ​​acceleration chip, and the communication stream runs in the communication computing unit of the AI ​​acceleration chip.

[0057] Optionally, the general-purpose computing unit and the communication computing unit are dedicated hardware components within the AI ​​acceleration chip that run the computational and communication streams respectively. The general-purpose computing unit performs pure computational operations, while the communication computing unit performs data transmission operations. The communication and computational streams are isolated by hardware, eliminating resource contention and enabling true parallelism between computation and communication.

[0058] Optionally, the AI ​​acceleration chip can monitor the status of each functional module in the computing and communication streams in real time, and determine whether the startup conditions are met based on the execution dependencies between modules. When the functional module to be started meets the startup conditions, the AI ​​acceleration chip immediately starts executing the functional module.

[0059] Optionally, within a short time after starting the computation, the host can sequentially distribute all the functional modules of the Transformer layer to the computation stream and the communication stream. The timing relationship of the execution of each functional module in the computation stream and the communication stream is as follows: Figure 3 As shown.

[0060] Optionally, the functional module to be started in the computation flow is the next computation module adjacent to the currently executed computation module, and the functional module to be started in the communication flow is the next communication module adjacent to the currently executed communication module.

[0061] Optionally, the completion of the currently executing functional module does not guarantee the immediate commencement of the next adjacent functional module within the same execution flow. Continuing the previous example, in Figure 3 In the first data stream, after the distribution communication module 1 finishes execution, it cannot immediately start the merging communication module 1. Instead, it must wait until the hybrid expert sparse computing module 1 finishes execution before it can be determined that the conditions for starting the merging communication module 1 are met.

[0062] Optionally, using an AI acceleration chip, the currently executing computational module in the computational stream and the currently executing communication module in the communication stream can be determined based on the execution status of the functional modules in the computational stream and the communication stream, respectively. Based on the order in which the functional modules in the computational stream and the communication stream are issued, the functional modules to be started in the computational stream and the communication stream can be determined respectively.

[0063] Optionally, using an AI acceleration chip, the execution status of functional modules in the computation and communication flows, as well as the execution dependencies between these modules, is monitored to detect functional modules to be started. When a functional module is detected to meet the startup conditions, it is started and executed. This can include: Using an AI acceleration chip, based on the execution status of functional modules in the computation and communication streams and the execution dependencies between these modules, it is determined whether any of the following conditions are met: The functional module to be started is any functional module other than the pre-payment calculation module, and the pre-payment functional module in the Transformer layer where the functional module to be started is located has been executed. The functional module to be started is the pre-payment calculation module, and the functional module to be started receives the output data sent by the preceding Transformer layer; If the AI ​​acceleration chip is used, the function module to be started is determined to meet the startup conditions, and the function module is started and executed. If not, the system will continue to monitor the process using an AI acceleration chip until it determines that the functional module to be started meets the startup conditions, at which point the functional module will be started and executed.

[0064] The technical solution of this invention involves dividing the input data of a large AI model into two data streams on the host side, and determining the functional module order, grouping dependencies, and execution dependencies of each data stream based on the Transformer layer architecture of the large AI model. On the host side, based on the functional module order and grouping dependencies of each data stream, the functional modules corresponding to the two data streams are alternately ordered to obtain the delivery order of the functional modules, and each functional module is sequentially delivered to its corresponding computation stream or communication stream according to the delivery order. Finally, an AI acceleration chip is used to process the computation stream and communication stream... The system monitors the execution status of functional modules and the execution dependencies between them. It monitors the functional modules to be started in the computing and communication streams, and starts the execution of a functional module when the startup conditions are met. The computing stream runs in the general computing unit of the AI ​​acceleration chip, and the communication stream runs in the communication computing unit of the AI ​​acceleration chip. This isolation of computing and communication hardware resources ensures that communication and computing do not affect each other's performance. By alternately sending dual data streams, deep overlap between communication and computing can be achieved, hiding communication latency and effectively improving the overall throughput and resource utilization of the cluster.

[0065] Example 2 Figure 4 This is a flowchart of a parallel communication and computation method provided in Embodiment 2 of the present invention. This embodiment, based on the above embodiments, specifically illustrates the parallel communication and computation method. Figure 4 As shown, the method includes: S210. On the host side, the input data of the AI ​​large model is divided into two data streams, and the functional module ordering, grouping dependency relationship and execution dependency relationship of the single data stream are determined according to the Transformer layer architecture of the AI ​​large model.

[0066] Among them, determining the order of functional modules for a single data stream based on the Transformer layer architecture of the large AI model can include: On the host side, based on the Transformer layer architecture, the order of functional modules within a single Transformer layer and the order between each Transformer layer are determined; the order of functional modules within a single Transformer layer is as follows: pre-attention calculation module, core attention calculation module, distribution communication module, shared expert general calculation module, hybrid expert sparse calculation module, and merged communication module. On the host side, based on the interlayer order of each Transformer layer, the functional modules of each Transformer layer are sorted and stacked sequentially to determine the functional module order of a single data stream.

[0067] Optionally, the pre-attention computation module and the core attention computation module can be used to implement feature extraction of the attention mechanism; the distribution communication module and the merging communication module can be used to realize the transfer of lexical units between different devices; the shared expert general computation module and the hybrid expert sparse computation module realize sparse expert computation.

[0068] Optionally, the functional modules within each Transformer layer are ordered as follows: pre-attention calculation module, core attention calculation module, distribution communication module, shared expert general calculation module, hybrid expert sparse calculation module, and merging communication module. Based on the inter-layer order of each Transformer layer, the pre-attention calculation module of the next Transformer layer can be placed after the merging communication module of the previous Transformer layer to obtain the global functional module order.

[0069] S220. On the host side, based on the functional module sorting and grouping dependency of the single data stream, the functional modules are grouped in the functional module sorting corresponding to the two data streams respectively.

[0070] S230. Through the host end, the functional modules corresponding to the two data streams are sorted alternately in groups to obtain the distribution order of the functional modules.

[0071] In the order of the functional modules being distributed, the first two groups of functional modules belong to the same data stream.

[0072] The grouping dependencies include: the pre-attention calculation module is grouped separately; the core attention calculation module, the distribution communication module, and the shared expert general calculation module are grouped together; and the hybrid expert sparse calculation module and the merged communication module are grouped together.

[0073] Optionally, after grouping multiple modules, the order in which the functional modules within a group are deployed is consistent with the order of the functional modules within a single Transformer layer.

[0074] S240. Through the host, according to the order in which the functional modules are distributed, each functional module is sequentially distributed to the corresponding computing stream or communication stream.

[0075] S250: Through the AI ​​acceleration chip, based on the execution status of functional modules in the computing flow and communication flow and the execution dependencies between functional modules, the system monitors the functional modules to be started in the computing flow and communication flow, and starts the execution of the functional module when the detected functional module meets the start conditions.

[0076] The parallel communication and computation method further includes: When any communication module starts execution, the following operations are performed in parallel: The AI ​​acceleration chip calls the first communication computing unit to read the target data block from the full set of words to be sent, determine the parallel device to which each word in the target data block belongs, divide the target data block into multiple word groups, and generate quantity information packets and word data packets according to each word group and write them into the first circular buffer. The AI ​​acceleration chip calls the second communication computing unit to send corresponding lexical information to each parallel device according to the first circular buffer. When the lexical information sent by each parallel device is received, the lexical information is written into the second circular buffer. The lexical information includes the number of lexical units and the lexical data. The AI ​​acceleration chip calls the third communication computing unit to read word data in the second circular buffer, obtain the expert identifier corresponding to each word in the word data, and perform data type conversion and storage of the word based on the expert identifier.

[0077] Understandably, in order to further improve cluster throughput and resource utilization, the inventors creatively proposed a communication method that uses multi-level pipelined asynchronous data transmission in the communication module, based on the parallel execution of computing and communication modules.

[0078] Understandably, large AI models contain a large number of expert networks, and the storage and computing resources of a single AI acceleration chip are limited and cannot support all the experts. Therefore, expert parallelism is adopted, and different experts are assigned to different chips.

[0079] Optionally, in the scenario of parallel operation of large model experts, the hybrid expert sparse computing module needs to route the tokens from the current AI acceleration chip to the AI ​​acceleration chip of the specified expert for processing during computation. Different AI acceleration chips have independent memory spaces, and experts can only process the data in the memory of their local AI acceleration chip. Furthermore, the two AI acceleration chips cannot directly read data from each other. Therefore, data transmission between the two AI acceleration chips is required through inter-chip communication to transfer the tokens to the AI ​​acceleration chip of the expert before they can enter the expert computation process.

[0080] Therefore, before the hybrid expert sparse computing module executes, the word can be sent from the current AI acceleration chip to the target AI acceleration chip of the expert processing the word through the distribution communication module, in preparation for subsequent expert computing; after the expert computing is completed, the processed word results need to be transmitted from the AI ​​acceleration chip of the expert back to the original AI acceleration chip, and the data recovery of cross-device computing is realized by merging the communication module.

[0081] Optionally, whether it is the distribution communication module or the merging communication module, the actual process is to send the tokens calculated by the current AI acceleration chip to the target AI acceleration chip corresponding to each token. In fact, the calculation or communication operations are performed synchronously within each AI acceleration chip.

[0082] Optionally, during data transmission or data retrieval, communication modules need to perform data packaging, data transmission, and data parsing. This invention uses multiple independent dedicated hardware resources within the AI ​​acceleration chip to perform data packaging, data transmission, and data parsing operations in parallel, thereby realizing a multi-level pipelined asynchronous data transmission communication method.

[0083] Figure 5 As a timing diagram for an optional multi-stage pipelined asynchronous data transmission, such as Figure 5 As shown, different grayscale backgrounds represent the first, second, and third communication computing units that can be invoked in the AI ​​acceleration chip. It can be seen that the first communication computing unit only performs data packaging operations, the second only performs data transmission operations, and the third only performs data parsing operations. After completing the packaging operation of target data block 1, the first communication computing unit immediately executes the packaging operation of target data block 2. Simultaneously, the second communication computing unit synchronously executes the data transmission operation of target data block 1. After completing the data transmission operation of target data block 1, if target data block 2 has already been packaged, the second communication computing unit immediately executes the transmission operation of target data block 2. Simultaneously, the third communication computing unit synchronously executes the data parsing operation of target data block 1, and the first communication computing unit synchronously executes the packaging operation of target data block 3. This mainly describes how the multi-level pipelined asynchronous data transmission of each communication computing unit is implemented through the full processing flow of target data block 1. When there are multiple target data blocks, they only need to be processed sequentially according to the above processing order, which will not be elaborated here.

[0084] Optionally, the full set of terms to be sent currently refers to the complete set of term data that needs to be transmitted between devices when the current communication module is executed.

[0085] Optionally, parallel devices can refer to AI acceleration chips that participate in expert parallel computing in the cluster. Each device has a unique identifier, and the expert network is distributed and deployed on different parallel devices. The parallel device corresponding to a word can refer to the AI ​​acceleration chip where the expert who processes that word is located.

[0086] Optionally, when the communication module starts execution, it actually transmits data to a single data stream. For a single data stream, the data dimension of the full set of tokens to be sent can be represented as [token_num1, hidden_size1], where token_num1 represents the number of tokens in the full set of tokens, and hidden_size1 represents the length of the feature vector corresponding to each token in the full set of tokens.

[0087] Optionally, through the AI ​​acceleration chip, multiple independent dedicated hardware resources on the chip can be called to perform data packaging, data transmission and data parsing operations in parallel in units of data blocks. Among them, data transmission includes two steps: data sending and data receiving.

[0088] Optionally, the dedicated hardware resources may specifically include a first communication computing unit, a second communication computing unit, and a third communication computing unit, wherein the first communication computing unit, the second communication computing unit, and the third communication computing unit are communication computing units that process communication streams in parallel.

[0089] Optionally, by using an AI acceleration chip, the first communication computing unit can be invoked to read consecutive words from the currently untransmitted first word in the full set of words to be sent, according to a pre-set number of segments. The number of words in the target data block is equal to the preset number of segments. If the number of remaining words is less than the preset number of segments, the last target data block is formed according to the actual number of remaining words, thereby realizing the streaming splitting of large blocks of data.

[0090] Optionally, the AI ​​acceleration chip can call the first communication computing unit to divide each word in the target data block into multiple word groups according to the parallel device to which they belong, and generate quantity information packets and word data packets according to each word group and write them into the first circular buffer.

[0091] Optionally, the quantity information packet can refer to a data packet used to inform the parallel device of the number of tokens to be received, and the token data packet can refer to a data packet containing the actual token content.

[0092] Optionally, through the AI ​​acceleration chip, the first communication computing unit can be invoked to classify all words in the current target data block according to the target parallel device number corresponding to each word, so that words belonging to the same parallel device form a word group. Then, according to each word group, the number of words in each group is counted and a quantity information packet is generated. At the same time, the grouped word data is organized to generate a word data packet, and the quantity information packet and word data packet are written sequentially into the first circular buffer.

[0093] Optionally, the first circular buffer is a global storage pre-allocated by the first communication computing unit during initialization. It is managed according to the circular storage. The first circular buffer adopts a first-in-first-out strategy, and the head and tail of the queue in the first circular buffer are tracked by maintaining read pointers and write pointers.

[0094] Optionally, the AI ​​acceleration chip can call the second communication computing unit to send corresponding lexical information to each parallel device according to the first circular buffer, and when the lexical information sent by each parallel device is received, the lexical information is written into the second circular buffer; wherein, the lexical information includes the number of lexical elements and the lexical data.

[0095] Optionally, through the AI ​​acceleration chip, the second communication computing unit can be invoked to read the quantity information packets and word data packets to be sent sequentially from the first circular buffer. Based on the quantity information packets, word data packets, and the parallel device identifiers corresponding to the data packets, word information is obtained and sent to the corresponding devices. At the same time, when the second communication unit receives word information sent by the parallel devices, it sequentially buffers the word information in the second circular buffer according to the identifier of the device from which the word information originates.

[0096] Optionally, the second circular buffer is a global storage pre-allocated by the second communication computing unit during initialization. It is managed according to the circular storage method. The second circular buffer adopts a first-in-first-out strategy and tracks the head and tail of the queue in the second circular buffer by maintaining read pointers and write pointers.

[0097] Optionally, the AI ​​acceleration chip can call the third communication computing unit to read word data in the second circular buffer, obtain the expert identifier corresponding to each word in the word data, and perform data type conversion and storage of the word based on the expert identifier.

[0098] The technical solution of this invention involves dividing the input data of a large AI model into two data streams on the host side. Based on the Transformer layer architecture of the large AI model, the functional module order, grouping dependencies, and execution dependencies of each data stream are determined. On the host side, based on the functional module order and grouping dependencies of each data stream, the functional modules are grouped within the functional module order corresponding to the two data streams. The functional modules of the two data streams are then alternately sorted by group to obtain the delivery order of the functional modules. Finally, based on the delivery order, each functional module is sequentially delivered to its corresponding computation or communication stream, and then processed by an AI acceleration chip. Based on the execution status of functional modules in the computation and communication flows and the execution dependencies between these modules, the system monitors the functional modules to be started in both flows. When a functional module meets the startup conditions, it is started. This approach allows for grouping functional modules according to dependencies, reducing flow control complexity and improving execution stability. A single data stream is started first to ensure the system quickly enters a stable operating state, followed by alternating data streams. This ensures that one communication stream is available while the other is computed, maximizing parallel overlap. The communication process employs a three-stage pipelined asynchronous transmission, significantly improving data transfer efficiency between devices. The two data streams achieve parallel communication and computation, effectively improving the overall throughput and resource utilization of the cluster.

[0099] Example 3 Figure 6 This is a flowchart of a multi-level pipelined asynchronous data transmission communication method provided in Embodiment 3 of the present invention. Based on the above embodiments, this embodiment specifically illustrates a method for multiple communication computing units to execute multi-level pipelined asynchronous data transmission communication in parallel. For example... Figure 6 As shown, the method executes multi-level pipelined asynchronous data transmission and communication in parallel through steps S310, S320, and S330: S310. Through the AI ​​acceleration chip, the first communication computing unit is called to read the target data block from the full set of words to be sent, determine the parallel device to which each word in the target data block belongs, divide the target data block into multiple word groups, generate quantity information packets and word data packets according to each word group, and write them into the first circular buffer.

[0100] Specifically, the AI ​​acceleration chip invokes the first communication computing unit to read the target data block from the current full set of words to be sent, and determines the parallel device to which each word in the target data block belongs, including: Using the AI ​​acceleration chip, the first communication computing unit is invoked to read the target data block from the full vocabulary according to the preset number of segments, and, depending on the type of communication module currently being executed, performs one of the following: By using the AI ​​acceleration chip, the first communication computing unit is invoked. If it is determined that the communication module currently being executed is the distribution communication module, the corresponding expert identifier is calculated for each word in the target data block. Based on the mapping relationship between the expert identifier and the parallel device, the parallel device corresponding to each word is determined, and the mapping relationship between each word and the expert identifier is stored in the shared memory of the parallel device. By using the AI ​​acceleration chip, the first communication computing unit is invoked. If it is determined that the communication module currently being executed is a merged communication module, then the parallel device corresponding to each word is determined based on the parallel device from which each word in the target data block originates during the execution of the distribution communication module.

[0101] Optionally, based on a preset number of segments, target data blocks of the same size as the number of segments can be read sequentially from the beginning of the full set of words. The above data packaging operation can be performed on each target data block sequentially. If the remaining words are less than the number of segments, the remaining words can be packaged as the last target data block.

[0102] Optionally, the expert identifier can refer to the number information used to uniquely identify a certain expert, indicating which expert should process the lexical unit.

[0103] It is understandable that if the communication module currently being executed is the distribution communication module, then the expert dimension information corresponding to the full set of words to be sent can be calculated. Based on the expert dimension information, the expert identifier that each word needs to be sent can be determined.

[0104] In an optional example, if the expert dimension information expert_num=8, it means that there are 8 experts to choose from, from expert0 to expert7. For word A in the target data block, when its expert dimension information is represented as [1, 0, 0, 0, 0, 0, 1, 1], it indicates that word A needs to be sent to the three experts, expert0, expert6 and expert7.

[0105] Optionally, each AI acceleration chip can carry different experts. A mapping relationship is pre-established between expert identifiers and AI acceleration chips. For example, AI acceleration chip A can carry expert0-expert3, and AI acceleration chip B can carry expert4-expert7. Therefore, after determining the expert identifiers corresponding to the words based on the expert dimension information, the parallel devices corresponding to the words can be determined based on the correspondence between the expert identifiers and the parallel devices.

[0106] It is understandable that when the merged communication module starts execution, it actually returns the words to the original AI acceleration chip. Therefore, based on the source parallel device information recorded by each word during the previous distribution communication module execution, the device to which each word needs to be returned is determined, and the word is moved from the current expert's device back to the original device, thus completing the data recycling.

[0107] The advantage of this setup is that it ensures the correctness and efficiency of word data transmission in expert parallel scenarios, while avoiding redundant calculations and improving overall reasoning efficiency.

[0108] Specifically, the target data block is divided into multiple word groups. Based on each word group, a quantity information packet and a word data packet are generated and written into the first circular buffer, including: The AI ​​acceleration chip calls the first communication computing unit to divide the words belonging to the same parallel device into the same word group according to the parallel device corresponding to each word in the target data block, and sorts the words in each word group according to the word identifier. The AI ​​acceleration chip calls the first communication computing unit to convert the words in each word group into the first data format required for data transmission, and performs deduplication based on word identifiers. The AI ​​acceleration chip calls the first communication computing unit to determine the number of words to be received by each parallel device and the word data based on the deduplicated word grouping, and generates quantity information packets and word data packets, and writes the quantity information packets and word data packets into the first circular buffer in sequence.

[0109] Optionally, a lexical identifier can refer to a number used to uniquely identify a lexical. In lexical grouping, lexicals can be sorted in ascending order of their lexical identifiers.

[0110] Optionally, the first data format can refer to a data format optimized for data transmission, such as FP8. The first data format has the characteristics of low bandwidth consumption and fast transmission, which is different from the data format used for computing.

[0111] Optionally, deduplication can refer to detecting duplicate transmissions of the same term, ensuring that the same term is sent only once to the same parallel device, avoiding redundant transmission and reducing communication load. Specifically, deduplication includes: when duplicate term identifiers are identified, only one term marked with the duplicate identifier is retained; after sorting the terms within a term group, deduplication can be performed quickly based on the sorting results.

[0112] Optionally, the AI ​​acceleration chip calls the first communication computing unit to convert the word data in the currently processed word group from the computing format to a first data format suitable for high-speed transmission, so as to reduce the data bit width and reduce the transmission bandwidth occupation, and then perform deduplication processing on the words in the word group according to the word identifier.

[0113] Optionally, the AI ​​acceleration chip calls the first communication computing unit to determine the number of words to be received by the parallel device corresponding to each word group based on the deduplicated word grouping, and generates a quantity information packet based on the quantity information. At the same time, the deduplicated and format-converted word data is organized into a word data packet. The quantity information packet is written into the first circular buffer in the order of quantity information packet first and word data packet second, and the buffer write pointer is updated synchronously to provide ready data for the subsequent data transmission stage.

[0114] Optionally, in the first circular buffer, the quantity information packet may include the number of tokens to be sent to each parallel device in the target data block, and the token data packet may include the token data to be sent to each parallel device.

[0115] The advantage of this setup is that it enables the standardization and lightweighting of transmitted data, providing a stable, orderly, and efficient data source for subsequent multi-level asynchronous pipeline transmission, thereby achieving high-throughput, low-overhead expert parallel communication.

[0116] S320: Through the AI ​​acceleration chip, the second communication computing unit is called to send the corresponding lexical information to each parallel device according to the first circular buffer, and when the lexical information sent by each parallel device is received, the lexical information is written into the second circular buffer.

[0117] The lexical information includes the number of lexical units and the data of lexical units.

[0118] Optionally, the first circular buffer and the second circular buffer can be the same size.

[0119] Specifically, the AI ​​acceleration chip invokes the second communication computing unit to send corresponding lexical information to each parallel device according to the first circular buffer. Upon receiving lexical information from each parallel device, the lexical information is written into the second circular buffer, which may include: The AI ​​acceleration chip is used to call the second communication computing unit to obtain the source address of the target quantity information packet and the target word data packet to be sent based on the first circular buffer. The AI ​​acceleration chip calls the second communication computing unit to determine the number of tokens corresponding to each parallel device based on the target quantity information packet, and triggers the transmission of token information based on the number of tokens corresponding to each parallel device and the source address of the target token data packet. The AI ​​acceleration chip calls the second communication computing unit. When it receives lexical information sent by each parallel device, it writes the received lexical information into the second circular buffer according to the device identifier of each parallel device.

[0120] Optionally, the AI ​​acceleration chip calls the second communication computing unit to obtain the target quantity information packet to be sent this time according to the current read pointer position of the first circular buffer, determine the number of tokens to be received by each parallel device, and obtain the source address of the target token data packet in global memory to prepare for direct hardware data transmission.

[0121] Optionally, by calling the second communication computing unit through the AI ​​acceleration chip, a transmission command can be sent to the communication hardware in the AI ​​acceleration chip according to the number of lexical units corresponding to each parallel device, the source address of the target lexical data packet, and the device identifier of each parallel device. This triggers the hardware transmission process of lexical information, thereby sending the lexical information composed of the number of lexical units and the lexical data units to each parallel device through the communication hardware.

[0122] Optionally, the AI ​​acceleration chip itself also serves as a parallel device for other AI acceleration chips in the cluster, receiving, parsing, and storing the lexical information sent by the parallel devices for use by the AI ​​acceleration chip after starting the hybrid expert sparse computing module.

[0123] Optionally, the AI ​​acceleration chip calls the second communication computing unit. When it receives lexical information sent by other parallel devices, it sorts the lexical information in ascending order of the device identifier according to the sender device identifier of each lexical information. Based on the sorting result, it writes the lexical information into the second circular buffer sequentially, starting from the first sorted lexical information.

[0124] S330, through the AI ​​acceleration chip, calls the third communication computing unit to read word data in the second circular buffer, obtain the expert identifier corresponding to each word in the word data, and perform data type conversion and storage of the word according to the expert identifier.

[0125] Specifically, the AI ​​acceleration chip invokes the third communication computing unit to read word metadata from the second circular buffer, obtains the expert identifier corresponding to each word in the word metadata, and performs data type conversion and storage of the word based on the expert identifier. This may include: The AI ​​acceleration chip calls the third communication computing unit to read word data in the second buffer based on the number of words in the second buffer, and determines the expert identifier corresponding to each word based on the shared memory of the parallel device. The AI ​​acceleration chip calls the third communication computing unit to convert the word into the second data format required for expert computing based on the expert identifier corresponding to the word, and then stores the converted word in the target storage area corresponding to the expert identifier.

[0126] Optionally, during the execution of the distribution communication module, the correspondence between each word and the expert identifier has been stored in the shared memory of the parallel device. Therefore, during the data parsing stage, the expert identifier corresponding to each word can be directly read from the shared memory, thereby effectively reducing the amount of computation and improving the data parsing efficiency.

[0127] Optionally, data type conversion can refer to converting the token from the transmission format to the format required by the expert calculation. For example, the token can be converted from the transmission format FP8 to the format required by the expert calculation, such as FP16 or FP32, thereby ensuring the correctness of the calculation.

[0128] Optionally, in the AI ​​acceleration chip, a designated storage area can be allocated to each expert, so that the format-converted lexical units are stored in the target storage area corresponding to the expert identifier according to the expert identifier corresponding to the lexical unit.

[0129] The technical solution of this invention uses an AI acceleration chip to call the first communication computing unit, the second communication computing unit, and the third communication computing unit respectively to realize a multi-level pipelined asynchronous data transmission communication method, which can significantly improve the communication speed. Multiple dedicated hardware components perform data packaging, data transmission, and data parsing respectively without blocking each other, without occupying computing hardware resources, and without affecting the computing performance of large AI models. The use of a dual-ring buffer to achieve decoupling of sending and receiving can effectively improve pipeline efficiency and throughput.

[0130] Example 4 Figure 7 This is a schematic diagram of a parallel communication and computing device provided in Embodiment 4 of the present invention. Figure 7 As shown, the device includes: a sorting determination module 410, a function module issuing module 420, and a function module activation module 430.

[0131] The sorting determination module 410 is used to divide the input data of the AI ​​large model into two data streams through the host, and determine the sorting, grouping dependency and execution dependency of the functional modules of the single data stream according to the Transformer layer architecture of the AI ​​large model; wherein, the functional modules include computing modules and communication modules.

[0132] The function module distribution module 420 is used to, through the host, alternately sort the function modules corresponding to the two data streams according to the function module sorting and grouping dependency of the single data stream, obtain the distribution order of the function modules, and distribute each function module to the corresponding computing stream or communication stream in sequence according to the distribution order of the function modules.

[0133] The function module startup module 430 is used to monitor the function modules to be started in the computing flow and communication flow through the AI ​​acceleration chip, based on the execution status of the function modules in the computing flow and communication flow and the execution dependencies between the function modules, and to start and execute the function module when the function module to be started meets the startup conditions; wherein, the computing flow runs in the general computing unit of the AI ​​acceleration chip, and the communication flow runs in the communication computing unit of the AI ​​acceleration chip.

[0134] The technical solution of this invention involves dividing the input data of a large AI model into two data streams on the host side, and determining the functional module order, grouping dependencies, and execution dependencies of each data stream based on the Transformer layer architecture of the large AI model. On the host side, based on the functional module order and grouping dependencies of each data stream, the functional modules corresponding to the two data streams are alternately ordered to obtain the delivery order of the functional modules, and each functional module is sequentially delivered to its corresponding computation stream or communication stream according to the delivery order. Finally, an AI acceleration chip is used to process the computation stream and communication stream... The system monitors the execution status of functional modules and the execution dependencies between them. It monitors the functional modules to be started in the computing and communication streams, and starts the execution of a functional module when the startup conditions are met. The computing stream runs in the general computing unit of the AI ​​acceleration chip, and the communication stream runs in the communication computing unit of the AI ​​acceleration chip. This isolation of computing and communication hardware resources ensures that communication and computing do not affect each other's performance. By alternately sending dual data streams, deep overlap between communication and computing can be achieved, hiding communication latency and effectively improving the overall throughput and resource utilization of the cluster.

[0135] Based on the above embodiments, the sorting determination module 410 can be specifically used for: On the host side, based on the Transformer layer architecture, the order of functional modules within a single Transformer layer and the order between each Transformer layer are determined; the order of functional modules within a single Transformer layer is as follows: pre-attention calculation module, core attention calculation module, distribution communication module, shared expert general calculation module, hybrid expert sparse calculation module, and merged communication module. On the host side, based on the interlayer order of each Transformer layer, the functional modules of each Transformer layer are sorted and stacked sequentially to determine the functional module order of a single data stream.

[0136] Based on the above embodiments, the function module distribution module 420 can be specifically used for: On the host side, based on the functional module sorting and grouping dependency of the single data stream, the functional modules are grouped in the functional module sorting corresponding to the two data streams respectively; On the host side, the functional modules corresponding to the two data streams are sorted alternately by group to obtain the distribution order of the functional modules; among them, the first two groups of functional modules belong to the same data stream in the distribution order of the functional modules. The grouping dependencies include: the pre-attention calculation module is grouped separately; the core attention calculation module, the distribution communication module, and the shared expert general calculation module are grouped together; and the hybrid expert sparse calculation module and the merged communication module are grouped together.

[0137] Based on the above embodiments, a data communication module may also be included, which may specifically include a data packaging submodule, a data sending submodule, and a data parsing submodule; when any communication module starts execution, the data packaging submodule, the data sending submodule, and the data parsing submodule are executed in parallel; The data packaging submodule is used to call the first communication computing unit through the AI ​​acceleration chip to read the target data block from the full set of words to be sent, determine the parallel device to which each word in the target data block belongs, divide the target data block into multiple word groups, generate quantity information packets and word data packets according to each word group, and write them into the first circular buffer. The data transmission submodule is used to call the second communication computing unit through the AI ​​acceleration chip, and send the corresponding lexical information to each parallel device according to the first circular buffer. When the lexical information sent by each parallel device is received, the lexical information is written into the second circular buffer. The lexical information includes the number of lexical elements and the lexical data. The data parsing submodule is used to call the third communication computing unit through the AI ​​acceleration chip to read word metadata in the second circular buffer, obtain the expert identifier corresponding to each word in the word metadata, and perform data type conversion and storage of the word based on the expert identifier.

[0138] Based on the above embodiments, the data packaging submodule can be used for: Using the AI ​​acceleration chip, the first communication computing unit is invoked to read the target data block from the full vocabulary according to the preset number of segments, and, depending on the type of communication module currently being executed, performs one of the following: By using the AI ​​acceleration chip, the first communication computing unit is invoked. If it is determined that the communication module currently being executed is the distribution communication module, the corresponding expert identifier is calculated for each word in the target data block. Based on the mapping relationship between the expert identifier and the parallel device, the parallel device corresponding to each word is determined, and the mapping relationship between each word and the expert identifier is stored in the shared memory of the parallel device. By using the AI ​​acceleration chip, the first communication computing unit is invoked. If it is determined that the communication module currently being executed is a merged communication module, then the parallel device corresponding to each word is determined based on the parallel device from which each word in the target data block originates during the execution of the distribution communication module.

[0139] Based on the above embodiments, the data packaging submodule can be specifically used for: The AI ​​acceleration chip calls the first communication computing unit to divide the words belonging to the same parallel device into the same word group according to the parallel device corresponding to each word in the target data block, and sorts the words in each word group according to the word identifier. The AI ​​acceleration chip calls the first communication computing unit to convert the words in each word group into the first data format required for data transmission, and performs deduplication based on word identifiers. The AI ​​acceleration chip calls the first communication computing unit to determine the number of words to be received by each parallel device and the word data based on the deduplicated word grouping, and generates quantity information packets and word data packets, and writes the quantity information packets and word data packets into the first circular buffer in sequence.

[0140] Based on the above embodiments, the data sending submodule can be specifically used for: The AI ​​acceleration chip is used to call the second communication computing unit to obtain the source address of the target quantity information packet and the target word data packet to be sent based on the first circular buffer. The AI ​​acceleration chip calls the second communication computing unit to determine the number of tokens corresponding to each parallel device based on the target quantity information packet, and triggers the transmission of token information based on the number of tokens corresponding to each parallel device and the source address of the target token data packet. The AI ​​acceleration chip calls the second communication computing unit. When it receives lexical information sent by each parallel device, it writes the received lexical information into the second circular buffer according to the device identifier of each parallel device.

[0141] Based on the above embodiments, the data parsing submodule can be specifically used for: The AI ​​acceleration chip calls the third communication computing unit to read word data in the second buffer based on the number of words in the second buffer, and determines the expert identifier corresponding to each word based on the shared memory of the parallel device. The AI ​​acceleration chip calls the third communication computing unit to convert the word into the second data format required for expert computing based on the expert identifier corresponding to the word, and then stores the converted word in the target storage area corresponding to the expert identifier.

[0142] The communication and computing parallel device provided in the embodiments of the present invention can execute the communication and computing parallel method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0143] Example 5 Figure 8 A schematic diagram of an electronic device 10, which can be used to implement embodiments of the present invention, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0144] like Figure 8 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0145] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0146] Processor 11 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, central processing unit (CPU), graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as the communication and computation parallel method described in the embodiments of the present invention. That is: On the host side, the input data of the large AI model is divided into two data streams, and the functional modules, grouping dependencies, and execution dependencies of each data stream are determined according to the Transformer layer architecture of the large AI model. The functional modules include computational modules and communication modules. On the host side, based on the functional module sorting and grouping dependency of the single data stream, the functional modules corresponding to the two data streams are alternately sorted to obtain the delivery order of the functional modules. Then, according to the delivery order of the functional modules, each functional module is sequentially delivered to the corresponding computing stream or communication stream. The AI ​​acceleration chip monitors the functional modules to be started in the computing and communication streams based on their execution status and the execution dependencies between them. When a functional module to be started meets the startup conditions, it is started and executed. The computing stream runs in the general computing unit of the AI ​​acceleration chip, and the communication stream runs in the communication computing unit of the AI ​​acceleration chip.

[0147] In some embodiments, the communication and computation parallel method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or mounted on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the communication and computation parallel method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to execute the communication and computation parallel method by any other suitable means (e.g., by means of firmware).

[0148] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0149] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0150] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0151] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0152] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0153] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0154] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0155] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for parallel communication and computation, characterized in that, This is executed by the host computer in conjunction with an AI acceleration chip, including: On the host side, the input data of the large AI model is divided into two data streams, and the functional modules, grouping dependencies, and execution dependencies of each data stream are determined according to the Transformer layer architecture of the large AI model. The functional modules include computational modules and communication modules. On the host side, based on the functional module sorting and grouping dependency of the single data stream, the functional modules corresponding to the two data streams are alternately sorted to obtain the delivery order of the functional modules. Then, according to the delivery order of the functional modules, each functional module is sequentially delivered to the corresponding computing stream or communication stream. The AI ​​acceleration chip monitors the functional modules to be started in the computation and communication streams based on their execution status and the execution dependencies between them. When a functional module meets the startup conditions, it is started and executed. The computation stream runs in the general-purpose computing unit of the AI ​​acceleration chip, and the communication stream runs in the communication computing unit of the AI ​​acceleration chip. The general-purpose computing unit and the communication computing unit are dedicated hardware within the AI ​​acceleration chip that run the computation stream and the communication stream, respectively. Based on the Transformer layer architecture of the large AI model, the functional modules of a single data stream are ordered, including: On the host side, based on the Transformer layer architecture, determine the order of functional modules within a single Transformer layer and the order between each Transformer layer. On the host side, based on the inter-layer order of each Transformer layer, the functional modules of each Transformer layer are sorted and stacked sequentially to determine the functional module order of a single data stream. The parallel communication and computation method further includes: When any communication module starts execution, the following operations are performed in parallel: The AI ​​acceleration chip is used to call the first communication computing unit to read the target data block from the full set of words to be sent. The parallel device to which each word in the target data block belongs is determined, and the target data block is divided into multiple word groups. Based on each word group, the number of words in each group is counted, and a quantity information packet is generated. Simultaneously, the grouped word data is organized to generate a word data packet. The quantity information packet and the word data packet are then written sequentially into the first circular buffer. Here, the parallel device refers to the AI ​​acceleration chip participating in expert parallel computing in the cluster; the quantity information packet is the data packet used to inform the parallel device of the number of words to be received. The AI ​​acceleration chip calls the second communication computing unit to send corresponding lexical information to each parallel device according to the first circular buffer. When the lexical information sent by each parallel device is received, the lexical information is written into the second circular buffer. The lexical information includes the number of lexical units and the lexical data. The AI ​​acceleration chip calls the third communication computing unit to read word data in the second circular buffer, obtain the expert identifier corresponding to each word in the word data, and perform data type conversion and storage of the word based on the expert identifier.

2. The parallel communication and computation method according to claim 1, characterized in that, The functional modules within a single Transformer layer are ordered as follows: pre-attention calculation module, core attention calculation module, distribution and communication module, shared expert general calculation module, hybrid expert sparse calculation module, and merging communication module.

3. The parallel communication and computation method according to claim 2, characterized in that, Based on the functional module sorting and grouping dependencies of the single data stream, the functional modules corresponding to the two data streams are alternately sorted to obtain the delivery order of the functional modules, including: On the host side, based on the functional module sorting and grouping dependency of the single data stream, the functional modules are grouped in the functional module sorting corresponding to the two data streams respectively; On the host side, the functional modules corresponding to the two data streams are sorted alternately by group to obtain the distribution order of the functional modules; among them, the first two groups of functional modules belong to the same data stream in the distribution order of the functional modules. The grouping dependencies include: the pre-attention calculation module is grouped separately; the core attention calculation module, the distribution communication module, and the shared expert general calculation module are grouped together; and the hybrid expert sparse calculation module and the merged communication module are grouped together.

4. The parallel communication and computation method according to claim 1, characterized in that, The AI ​​acceleration chip is used to call the first communication computing unit to read the target data block from the full set of words to be sent, and to determine the parallel device to which each word in the target data block belongs, including: Using the AI ​​acceleration chip, the first communication computing unit is invoked to read the target data block from the full vocabulary according to the preset number of segments, and, depending on the type of communication module currently being executed, performs one of the following: By using the AI ​​acceleration chip, the first communication computing unit is invoked. If it is determined that the communication module currently being executed is the distribution communication module, the corresponding expert identifier is calculated for each word in the target data block. Based on the mapping relationship between the expert identifier and the parallel device, the parallel device corresponding to each word is determined, and the mapping relationship between each word and the expert identifier is stored in the shared memory of the parallel device. By using the AI ​​acceleration chip, the first communication computing unit is invoked. If it is determined that the communication module currently being executed is a merged communication module, then the parallel device corresponding to each word is determined based on the parallel device from which each word in the target data block originates during the execution of the distribution communication module.

5. The parallel communication and computation method according to claim 1, characterized in that, The target data block is divided into multiple word groups. Based on each word group, a quantity information packet and a word data packet are generated and written into the first circular buffer, including: The AI ​​acceleration chip calls the first communication computing unit to divide the words belonging to the same parallel device into the same word group according to the parallel device corresponding to each word in the target data block, and sorts the words in each word group according to the word identifier. The AI ​​acceleration chip calls the first communication computing unit to convert the words in each word group into the first data format required for data transmission, and performs deduplication based on word identifiers. The AI ​​acceleration chip calls the first communication computing unit to determine the number of words to be received by each parallel device and the word data based on the deduplicated word grouping, and generates quantity information packets and word data packets, and writes the quantity information packets and word data packets into the first circular buffer in sequence.

6. The parallel communication and computation method according to claim 1, characterized in that, The AI ​​acceleration chip invokes the second communication computing unit to send corresponding lexical information to each parallel device according to the first circular buffer. Upon receiving lexical information from each parallel device, the lexical information is written into the second circular buffer. The lexical information includes the number of lexical units and lexical data, including: The AI ​​acceleration chip is used to call the second communication computing unit to obtain the source address of the target quantity information packet and the target word data packet to be sent based on the first circular buffer. The AI ​​acceleration chip calls the second communication computing unit to determine the number of tokens corresponding to each parallel device based on the target quantity information packet, and triggers the transmission of token information based on the number of tokens corresponding to each parallel device and the source address of the target token data packet. The AI ​​acceleration chip calls the second communication computing unit. When it receives lexical information sent by each parallel device, it writes the received lexical information into the second circular buffer according to the device identifier of each parallel device.

7. The parallel communication and computation method according to claim 1, characterized in that, The AI ​​acceleration chip invokes the third communication computing unit to read word metadata from the second circular buffer, obtains the expert identifier corresponding to each word in the word metadata, and performs data type conversion and storage of the word based on the expert identifier, including: The AI ​​acceleration chip calls the third communication computing unit to read word data in the second buffer based on the number of words in the second buffer, and determines the expert identifier corresponding to each word based on the shared memory of the parallel device. The AI ​​acceleration chip calls the third communication computing unit to convert the word into the second data format required for expert computing based on the expert identifier corresponding to the word, and then stores the converted word in the target storage area corresponding to the expert identifier.

8. A parallel communication and computing device, characterized in that, This is executed by the host computer in conjunction with an AI acceleration chip, including: The sorting determination module is used to divide the input data of the large AI model into two data streams on the host side, and determine the sorting, grouping dependencies, and execution dependencies of the functional modules of each data stream according to the Transformer layer architecture of the large AI model; among them, the functional modules include computation modules and communication modules; The function module distribution module is used by the host to alternately sort the function modules corresponding to the two data streams according to the function module order and grouping dependency of the single data stream, obtain the distribution order of the function modules, and distribute each function module to the corresponding computing stream or communication stream in sequence according to the distribution order of the function modules. The function module startup module is used to monitor the function modules to be started in the computing flow and communication flow through the AI ​​acceleration chip, based on the execution status of the function modules in the computing flow and communication flow and the execution dependencies between the function modules. When the function module to be started meets the startup conditions, the module is started and executed. The computing flow runs in the general computing unit of the AI ​​acceleration chip, and the communication flow runs in the communication computing unit of the AI ​​acceleration chip. The general computing unit and the communication computing unit are dedicated hardware within the AI ​​acceleration chip that respectively run the computing flow and the communication flow. The sorting determination module is specifically used for: On the host side, based on the Transformer layer architecture, determine the order of functional modules within a single Transformer layer and the order between each Transformer layer. On the host side, based on the inter-layer order of each Transformer layer, the functional modules of each Transformer layer are sorted and stacked sequentially to determine the functional module order of a single data stream. The parallel communication and computing device further includes a data communication module, specifically comprising a data packaging submodule, a data sending submodule, and a data parsing submodule; when any communication module is started, the data packaging submodule, the data sending submodule, and the data parsing submodule are executed in parallel. The data packaging submodule is used to call the first communication computing unit through the AI ​​acceleration chip to read the target data block from the full set of words to be sent, determine the parallel device to which each word in the target data block belongs, divide the target data block into multiple word groups, count the number of words in each group according to each word group and generate a quantity information packet, and organize the grouped word data into a word data packet. The quantity information packet and the word data packet are written sequentially into the first circular buffer. Here, the parallel device refers to the AI ​​acceleration chip participating in expert parallel computing in the cluster; the quantity information packet refers to the data packet used to inform the parallel device of the number of words to be received. The data transmission submodule is used to call the second communication computing unit through the AI ​​acceleration chip, and send the corresponding lexical information to each parallel device according to the first circular buffer. When the lexical information sent by each parallel device is received, the lexical information is written into the second circular buffer. The lexical information includes the number of lexical elements and the lexical data. The data parsing submodule is used to call the third communication computing unit through the AI ​​acceleration chip to read word metadata in the second circular buffer, obtain the expert identifier corresponding to each word in the word metadata, and perform data type conversion and storage of the word based on the expert identifier.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the communication and computation parallel method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the communication and computation parallel method of any one of claims 1-7.

Images