Task processing method and device

Abstract

The invention provides a task processing method and device, and relates to the technical field of artificial intelligence. The task processing method comprises the following steps: splitting a target model, and inserting a data synchronization operator to obtain a plurality of model blocks; the plurality of model blocks are loaded into a plurality of acceleration cards, and every two acceleration cards are mutually connected; and in response to the obtained to-be-reasoned task, performing parallel calculation on the to-be-reasoned task based on the plurality of accelerator cards to obtain a reasoning result, the calculation including corresponding calculation of a model operator in the target model and corresponding calculation of a data synchronization operator. According to the method, through the cooperation of compiling, reasoning service and software and hardware combination of acceleration card hardware, calculation and data synchronization are achieved in a high-precision and low-delay mode, the accuracy is guaranteed, meanwhile, the synchronization cost in the model reasoning process is greatly reduced, and the synchronization time is shortened.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, specifically to a method and apparatus for processing a task. Background Technology

[0002] Currently, with the development of AI large model technology and the increasing scale of models, a single inference accelerator card can no longer accommodate the entire model, and it has become the norm for multiple accelerator cards to work together to infer a single model.

[0003] Since a single inference request involves multiple and various data synchronizations (AllReduce / Broadcast / AllGather), the performance of data collaboration between multiple GPUs is crucial.

[0004] However, due to communication delays, clock misalignments, and latency caused by software applications, multiple accelerator cards may experience issues such as clock asynchrony and inconsistent computation rhythms, leading to problems such as data synchronization deviations, excessively long synchronization waiting times, and low computation efficiency.

[0005] Traditional methods include software synchronization and hardware synchronization, but both have certain shortcomings: software synchronization has low accuracy (millisecond level) and high latency, which cannot meet the requirements of AI inference; traditional hardware synchronization requires dedicated hardware support, has a complex architecture, high cost, and only has simple synchronization functions. Summary of the Invention

[0006] Based on this, this application provides a task processing method and apparatus to achieve data synchronization with high accuracy, low cost and low latency.

[0007] According to one aspect of this application, a task processing method is proposed, comprising: splitting a target model and inserting data synchronization operators to obtain multiple model blocks; loading the multiple model blocks into multiple accelerator cards, wherein the multiple accelerator cards are interconnected in pairs; and, in response to the obtained inference task, performing parallel computation on the inference task based on the multiple accelerator cards to obtain inference results, wherein the computation includes corresponding computation of model operators in the target model and corresponding computation of data synchronization operators.

[0008] According to some embodiments, the target model is split and data synchronization operators are inserted to obtain multiple model blocks, including: performing tensor parallel split on the preset weight parameters of the target model to obtain the split results; extracting the computing nodes that need to be synchronized with data in the split results, and inserting data synchronization operators with corresponding functions at the positions of the computing nodes to obtain multiple model blocks.

[0009] According to some embodiments, any two accelerator cards among a plurality of accelerator cards are connected via a hardware interface.

[0010] According to some embodiments, in response to the acquired inference task, the inference task is computed in parallel based on multiple accelerator cards to obtain the inference result, including: in response to the acquired inference task, writing the inference task into the memory of multiple accelerator cards; based on multiple accelerator cards, executing the model operators and / or data synchronization operators in the corresponding model block in parallel, and exchanging data when executing the data synchronization operator, until traversing the operators in the corresponding model block, and outputting the computation result as the inference result.

[0011] According to some embodiments, based on multiple accelerator cards, model operators and / or data synchronization operators in the corresponding model block are executed in parallel, and data exchange is performed when the data synchronization operator is executed, until the operators in the corresponding model block are traversed, and the calculation result is output as the inference result, including: S1: Based on multiple accelerator cards, the current operator is extracted from the operators in the corresponding model block; S2: If the current operator is a model operator, the computing units built into multiple accelerator cards are called to complete the corresponding calculation and obtain the current calculation result; S3: If the current operator is a data synchronization operator, the synchronization computing units built into multiple accelerator cards are called to complete the corresponding calculation and obtain the current calculation result; S4: S1-S3 are repeated until multiple accelerator cards have traversed the operators in the corresponding model block, and the current calculation result of the last iteration is output as the inference result.

[0012] According to some embodiments, step S3 includes: S301: When the current operator is the first data synchronization operator, the first data stored by each of the multiple accelerator cards is obtained as the current data, and the memory space of the data from other accelerator cards is allocated based on the remaining memory space of each of the multiple accelerator cards to obtain the allocation result, wherein the first data synchronization operator includes the AllGather operator; S302: The synchronization calculation unit built into the multiple accelerator cards is called, and the current data of the corresponding accelerator card is read according to the pre-configured accelerator card data reading relationship as the first calculation result of the multiple accelerator cards; S303: The current data of the multiple accelerator cards is updated according to the first calculation result, and the first calculation result is stored in the corresponding location according to the allocation result; S304: Steps S302-S303 are repeated until the data in the multiple accelerator cards are completely consistent, and the completely consistent data is output as the current calculation result.

[0013] According to some embodiments, step S3 includes: S311: When the current operator is the second data synchronization operator, the second data stored by each of the multiple accelerator cards is obtained as the current data, and the memory space of the data from other accelerator cards is allocated based on the remaining memory space of each of the multiple accelerator cards to obtain the allocation result, wherein the second data synchronization operator includes the AllReduce operator; S312: The synchronous calculation unit built into the multiple accelerator cards is called to read the current data of the corresponding accelerator card according to the pre-configured accelerator card data reading relationship, and the corresponding calculation of the second data synchronization operator is performed according to the read current data and the second data stored by each of the multiple accelerator cards to obtain the second calculation result; S313: The current data of the multiple accelerator cards is updated according to the second calculation result, and the second calculation result is stored in the corresponding location according to the allocation result; S314: Steps S312-S313 are repeated until the data in the multiple accelerator cards are completely consistent, and the second calculation result of the last iteration is output as the current calculation result.

[0014] According to some embodiments, step S3 includes: S321: When the current operator is a third data synchronization operator, the third data stored by each of the multiple accelerator cards is divided into a first number of data blocks, and the data blocks are numbered, wherein the first number is the number of multiple accelerator cards, and the third data synchronization operator includes the AllReduce + Scatter operator; S322: The data block with the number of the first target number stored in each of the multiple accelerator cards is used as the current data, wherein the first target number is the number of the multiple accelerator cards; S323: The synchronous calculation unit built into the multiple accelerator cards is invoked to read the current data in the corresponding accelerator card according to the pre-configured accelerator card data reading relationship, as the second data to be processed; S324: The data block with the same number as the current data in the multiple accelerator cards is obtained as the first data to be processed; S325: According to the first data to be processed and the second data to be processed, the corresponding calculation of the third data synchronization operator is performed to obtain the third calculation result, and the current data of the multiple accelerator cards is updated according to the third calculation result; S326: The third calculation result is stored in the original storage location of the first data to be processed, and step S322 is repeated. S326: Continue until the third calculation result meets the preset conditions, where the preset conditions include completing the full accumulation of data blocks with the same number in multiple accelerator cards; S327: Call the synchronous calculation unit built into multiple accelerator cards, and read the latest data that meets the preset conditions in the corresponding accelerator card according to the data reading relationship of the accelerator cards, as the third data to be processed; S328: Store the third data to be processed in the target storage location of multiple accelerator cards, where the target storage location is the storage location of the data block with the same number as the third data to be processed in the memory space of multiple accelerator cards; S329: Repeat steps S327-S328 until the data in multiple accelerator cards are completely consistent, and output the completely consistent data as the current calculation result.

[0015] According to one aspect of this application, a task processing apparatus includes: a splitting module for splitting a target model and inserting data synchronization operators to obtain multiple model blocks; a loading module for loading the multiple model blocks into multiple accelerator cards, wherein the multiple accelerator cards are interconnected in pairs; and a calculation module for performing parallel calculations on the inference task based on the multiple accelerator cards in response to an acquired inference task to be obtained, to obtain an inference result, wherein the calculation includes corresponding calculations of model operators in the target model and corresponding calculations of data synchronization operators.

[0016] According to one aspect of this application, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more programs; and, when the one or more programs are executed by the one or more processors, causing the one or more processors to implement the method as described above.

[0017] According to one aspect of this application, a computer-readable medium is provided that stores a computer program or instructions thereon, which, when executed by a processor, implement the method as described above.

[0018] Through the embodiments provided in this application, model compilation is achieved by splitting the target model and inserting synchronization operators; the compiled target model is loaded onto multiple accelerator cards, user inference tasks are received, accelerator cards are invoked for execution, and inference results are returned after inference is completed, thus realizing model inference. This application achieves high-precision, low-latency synchronization of computation and data through the collaborative cooperation of compilation, inference services, and accelerator card hardware, ensuring correctness while significantly reducing synchronization costs and time during model inference. Attached Figure Description

[0019] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit this application.

[0020] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings, without exceeding the scope of protection claimed by this application.

[0021] Figure 1 A flowchart illustrating the task processing method provided in the embodiments of this application; Figure 2 One of the flowcharts provided in this application embodiment describes the process of splitting a target model and inserting a data synchronization operator to obtain multiple model blocks; Figure 3 The second flowchart provided in this application illustrates the process of splitting a target model and inserting a data synchronization operator to obtain multiple model blocks. Figure 4 This is one of the flowcharts provided in this application embodiment, which describes how a reasoning result is obtained by performing parallel computation on the reasoning task based on multiple accelerator cards in response to the acquired reasoning task. Figure 5 The second flowchart provided in this application describes how, in response to the acquired inference task, multiple accelerator cards perform parallel computation on the inference task to obtain the inference result. Figure 6 This is one of the data synchronization diagrams of multiple accelerator cards provided in the embodiments of this application; Figure 7 This is the second schematic diagram illustrating data synchronization of multiple accelerator cards provided in the embodiments of this application; Figures 8(a)-8(f) are the third type of data synchronization diagrams of multiple accelerator cards provided in the embodiments of this application; Figure 9 A block diagram of a task processing apparatus provided in the embodiments of this application; Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0022] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0023] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0024] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0025] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0026] It should be understood that although the terms first, second, third, etc., may be used herein to describe various components, these components should not be limited by these terms. These terms are used to distinguish one component from another. Therefore, the first component discussed below may be referred to as the second component without departing from the teachings of this application. As used herein, the term "and / or" includes all combinations of any one and more of the associated listed items.

[0027] For specific implementation details, please refer to the following examples.

[0028] Figure 1 A flowchart illustrating a task processing method provided in an embodiment of this application. Figure 1 As shown, the method includes steps S110-S130.

[0029] In step S110, the target model is split and a data synchronization operator is inserted to obtain multiple model blocks.

[0030] This step is mainly used to split the target model and insert synchronization operators.

[0031] The target model can be any model that performs the inference task, and this application does not impose any restrictions on it.

[0032] In the specific implementation process, the target model is split into multiple parts so that they can be distributed to multiple accelerator cards for parallel execution.

[0033] It is understandable that splitting the target model is not a simple division; the goal is to distribute the model across multiple accelerator cards for parallel computation. Therefore, splitting strategies such as Tensor Parallelism (TP) can be employed.

[0034] After the model is split, a data synchronization operator is inserted after the operators of the model itself that need to be synchronized. The multiple parts of the model after the insertion of the data synchronization operator are denoted as multiple model blocks.

[0035] According to the example implementation, the data synchronization operators include AllGataher, AllReduce, Broadcast, and other data synchronization operators.

[0036] According to the example embodiment, step S110 is implemented using an AI compiler. It is understood that the compiler only needs to split the model's data; the specific synchronization function implementation is transparent to the compiler, which only needs to insert data synchronization operators at the appropriate locations based on the model's functionality.

[0037] In step S120, multiple model blocks are loaded into multiple accelerator cards, wherein the multiple accelerator cards are interconnected in pairs.

[0038] Multiple model blocks are loaded onto various accelerator cards. Accelerator cards are interconnected in pairs, and multiple accelerator cards are fully interconnected, allowing each accelerator card to read and write data from other accelerator cards at will.

[0039] The accelerator card natively supports data synchronization operators such as AllGataher, AllReduce, and Broadcast.

[0040] During the loading of multiple model blocks, the original synchronization signal reception / transmission and data computation functions of the target model are encapsulated into atomic model operators. These model operators and the inserted data synchronization operators are collectively referred to as hardware operators. Hardware operators perform synchronization and computation simultaneously without waiting.

[0041] In step S130, in response to the acquired inference task, the inference task is computed in parallel based on multiple accelerator cards to obtain the inference result. The computation includes the corresponding computation of model operators in the target model and the corresponding computation of data synchronization operators.

[0042] After receiving the user's inference task, the inference request is sent to each accelerator card. The accelerator card is then called to automatically execute the loaded model block. When the original model operator is encountered, the calculation is automatically completed. When the data synchronization operator is encountered, the synchronization and calculation are automatically completed. After the entire model is executed, the inference result is returned.

[0043] It is important to emphasize that multiple accelerator cards execute in parallel during the computation of the model blocks.

[0044] It is understandable that in some embodiments, an inference call command is sent to multiple accelerator cards only once during the initialization of inference startup to fulfill the inference request.

[0045] According to the example embodiment, the compiled model block is loaded onto the accelerator card using an AI inference platform, and the user's inference question is received. The accelerator card is then invoked to execute the inference, and the result is returned after the inference is completed. In this embodiment, synchronization phase-locking is achieved through the collaborative cooperation between the compiler, the inference service, and the accelerator card hardware during the AI ​​inference process.

[0046] Through the embodiments provided in this application, model compilation is achieved by splitting the target model and inserting synchronization operators; the compiled target model is loaded onto multiple accelerator cards, user inference tasks are received, accelerator cards are invoked for execution, and inference results are returned after inference is completed, thus realizing model inference. This application achieves high-precision, low-latency synchronization of computation and data through the collaborative cooperation of compilation, inference services, and accelerator card hardware, ensuring correctness while significantly reducing synchronization costs and time during model inference.

[0047] According to some embodiments, refer to Figure 2 In step S110, the target model is split and a data synchronization operator is inserted to obtain multiple model blocks, which can be achieved through steps S210-S220.

[0048] In step S210, the preset weight parameters of the target model are split into tensors in parallel to obtain the splitting results.

[0049] The target model is split using TP (Tensor Parallelism), which splits a single tensor of the target model into multiple sub-tensors along a specified dimension, resulting in multiple views of the target model along the specified dimension, denoted as the splitting result.

[0050] According to the example implementation, the target model is split into TPs using a compiler dedicated to the accelerator card.

[0051] In step S220, the computing nodes that need to be synchronized in the splitting results are extracted, and the corresponding data synchronization operators are inserted at the positions of the computing nodes to obtain multiple model blocks.

[0052] Based on the model's functionality, extract the computation nodes in the split results that require data synchronization, insert data synchronization operators at the corresponding positions, and record multiple views with inserted data synchronization operators as multiple model blocks.

[0053] In one specific embodiment, refer to Figure 3 Using llama3 as the target model, the compiler splits the target model into four parts and inserts data synchronization operators. Figure 3 In this context, input, embeddings, RMSNorm, and Add are all model operators.

[0054] The steps include: 1) Use the compiler to perform TP segmentation on the Q (Query), K (Key), V (Value) weights and O_PROJ (Output Projection) weights of the target model input; 2) Insert the AllReduce synchronization operator after the calculation of the output projection (O_PROJ) is completed; 3) Perform the Add operation after completing the AllReduce operator to obtain multiple model blocks.

[0055] According to some embodiments, any two accelerator cards among a plurality of accelerator cards are connected via a hardware interface.

[0056] This application achieves full hardware synchronization through the interconnection of two accelerator cards.

[0057] In the specific reasoning process, for data synchronization operators such as AllGataher / AllReduce / Broadcast that require synchronization across multiple accelerator cards, the hardware-encapsulated synchronization operator interface is directly called, maintaining consistency with other general operators without any special processing.

[0058] According to some embodiments, refer to Figure 4 In step S130, in response to the acquired inference task, the inference task is computed in parallel based on multiple accelerator cards to obtain the inference result, which can be specifically achieved through steps S410-S420.

[0059] In step S410, in response to the acquired inference task, the inference task is written into the memory of multiple accelerator cards.

[0060] Users send a reasoning task, including the question to be reasoned, to the reasoning service (e.g., the reasoning platform). The reasoning service writes the input data into the memory of multiple accelerator cards and starts the reasoning process.

[0061] In step S420, based on multiple accelerator cards, the model operators and / or data synchronization operators in the corresponding model blocks are executed in parallel, and data exchange is performed when the data synchronization operator is executed, until the operators in the corresponding model blocks are traversed, and the calculation results are output as the inference results.

[0062] Multiple accelerator cards begin execution based on the model structure within the loaded model blocks. The execution includes: 1) Execute each operator in sequence (including model operators and data synchronization operators); 2) For the data synchronization operator, the hardware function module is invoked to complete the calculation; 3) Once the model execution is complete, the calculation results are returned as the inference results.

[0063] Then, the inference results are returned to the user through the inference service.

[0064] Understandably, throughout the process, the software providing inference services (such as the inference platform) is only responsible for writing input and reading output, and does not participate in the specific synchronization process, which can fully guarantee the performance of inference.

[0065] In one specific embodiment, refer to Figure 5 It provides inference services through a distributed AI inference platform, which includes four acceleration cards, namely: Acceleration Card 0, Acceleration Card 1, Acceleration Card 2, and Acceleration Card 3. Figure 5 In this model, input, embeddings, attention, and output are all model operators. Data synchronization operators include the AllReduce operator.

[0066] According to some embodiments, in step S420, based on multiple accelerator cards, the model operators and / or data synchronization operators in the corresponding model block are executed in parallel, and data exchange is performed when the data synchronization operator is executed, until the operators in the corresponding model block are traversed, and the calculation result is output as the inference result. Specifically, this can be achieved through steps S1-S4.

[0067] In step S1, the current operator is extracted from the operators in the corresponding model block based on multiple accelerator cards.

[0068] Based on the accelerator card's execution progress on the corresponding model block, extract the hardware operator that needs to be executed now, and denot it as the current operator.

[0069] Understandably, since the split model blocks are views of the original complete tensor based on the sub-tensors, multiple accelerator cards progress at the same pace during parallel computation; that is, multiple accelerator cards use the same current operator. (Refer to...) Figure 5 At the same time, the specific data used to execute the current operator is a part of the original complete data.

[0070] In step S2, when the current operator is a model operator, multiple built-in computing units of the accelerator card are invoked to complete the corresponding calculations and obtain the current calculation results.

[0071] If the current operator is a general operator (i.e., a model operator), then each of the multiple accelerator cards will call its built-in computing unit to complete the corresponding calculation and record the result as the current calculation result.

[0072] In step S3, if the current operator is a data synchronization operator, multiple built-in synchronization calculation units of the accelerator card are invoked to complete the corresponding calculations and obtain the current calculation results.

[0073] If the current operator is a data synchronization operator, then each of the multiple accelerator cards will call its built-in synchronous computing unit to complete the calculation and record the result as the current calculation result.

[0074] In step S4, S1-S3 are repeated until all accelerator cards have traversed the operators in the corresponding model blocks and output the current calculation result of the last iteration as the inference result.

[0075] Execute the next operator until all calculations are completed, and return the current calculation result of the last iteration as the inference result.

[0076] Understandably, the accelerator card has a built-in graph executor responsible for scheduling to execute the inference process based on the loaded model blocks. In practice, the accelerator, as the core processing unit on the accelerator card, completes the computation of the corresponding operators based on the scheduling of the graph executor. Specifically, for general operators (i.e., model operators), the graph executor calls the accelerator's built-in computation unit to complete the computation; for data synchronization operators, the graph executor calls the accelerator's built-in synchronization computation unit to complete the computation.

[0077] Furthermore, when each target model completes its inference task, an inference call command is sent to multiple accelerator cards only once during the initial inference startup.

[0078] In this embodiment of the application, all synchronization operations are completed within the accelerator card by calling the hardware interface in a single inference request. There is no need to notify the host server or schedule the inference service, which greatly reduces the latency of communication and scheduling, and improves the inference speed and the utilization of the accelerator card.

[0079] According to some embodiments, in step S3, when the current operator is a data synchronization operator, multiple built-in synchronization computing units of the accelerator card are called to complete the corresponding calculation and obtain the current calculation result. Specifically, this can be achieved through steps S301-S304.

[0080] In step S301, when the current operator is the first data synchronization operator, the first data stored by each of the multiple accelerator cards is obtained as the current data, and the memory space of the data from other accelerator cards is allocated based on the remaining memory space of each of the multiple accelerator cards to obtain the allocation result. The first data synchronization operator includes the AllGather operator.

[0081] The first data synchronization operator includes a data synchronization operator for performing data aggregation and synchronization. This application uses the AllGather operator as an example of the first data synchronization operator for illustration.

[0082] The AllGather operator is used to aggregate sub-Tensors (tensors) of the same shape distributed on multiple accelerator cards into a complete Tensor (containing the sub-Tensors of all devices) on each device. It is a core collective communication operator in distributed computing.

[0083] Each accelerator card completes its own calculation, and the calculation result is recorded as the first data. All data blocks storing the first data are READY (ready) and are recorded as the current data of each accelerator card.

[0084] At the same time, based on the remaining memory space of each of the multiple accelerator cards, space is allocated to store data blocks from other accelerator cards, and this allocation result is recorded.

[0085] In step S302, the synchronous computing units built into multiple accelerator cards are invoked, and the current data of the corresponding accelerator card is read according to the pre-configured accelerator card data reading relationship, which is used as the first calculation result of multiple accelerator cards.

[0086] The synchronous computing units built into multiple accelerator cards are invoked, and multiple accelerator cards begin reading and writing simultaneously.

[0087] During the actual read and write process, the current data of the corresponding accelerator card is read according to the pre-configured accelerator card data reading relationship to complete the data reading process.

[0088] Understandably, the data read relationships of the accelerator cards are configured during the process of splitting the target model and adding data synchronization operators. Furthermore, the accelerator card IDs are also assigned during this period.

[0089] According to the example implementation, the compiler determines the specific accelerator card to which the data is allocated during compilation and defines the synchronization process as the data reading relationship of the accelerator card.

[0090] In step S303, the current data of multiple accelerator cards are updated according to the first calculation result, and the first calculation result is stored in the corresponding location according to the allocation result.

[0091] The first calculation result of each accelerator card is used as the new current data, and the first calculation result is stored in the allocated location to complete the data writing process.

[0092] In step S304, steps S302-S303 are repeated until the data in multiple accelerator cards are completely consistent, and the completely consistent data is output as the current calculation result.

[0093] Repeat the data read and write process until the data in multiple accelerator cards is completely consistent, and output the completely consistent data as the current calculation result.

[0094] At this point, the entire AllGather process is complete, with the synchronization of tensors entirely within the hardware, requiring no involvement from the upper-layer inference platform. The traditional method involves multiple accelerator cards writing data separately to an inference service, which then aggregates the data and writes it back to each accelerator card. Because the inference service is required for synchronization, the latency is significantly higher (software read, write, and compute data via PCIe (Peripheral Component Interconnect Express, a high-speed serial computer expansion bus standard)), exceeding 10 times that of a purely hardware-based process.

[0095] To provide a more detailed description of the embodiments of this application, a specific embodiment is given. (Refer to...) Figure 6 In this embodiment, four acceleration cards are included, namely: acceleration card 0, acceleration card 1, acceleration card 2, and acceleration card 3.

[0096] Each accelerator card completes its own calculation, and its corresponding first data is ready: the first data stored in accelerator card 0 is data A, the first data stored in accelerator card 1 is data B, the first data stored in accelerator card 2 is data C, and the first data stored in accelerator card 3 is data D. These data are used as the current data for each accelerator card, and simultaneously, each accelerator card allocates space to store the other three data blocks.

[0097] The pre-configured data reading relationships of the accelerator cards include: accelerator card 0 reads data from accelerator card 3, accelerator card 1 reads data from accelerator card 0, accelerator card 2 reads data from accelerator card 1, and accelerator card 3 reads data from accelerator card 2.

[0098] Four accelerator cards begin reading and writing simultaneously, such as Figure 6 As shown in (a): Accelerator Card 0: Reads data D from Accelerator Card 3, uses data D as the new current data, and saves it to the locally allocated location; Accelerator Card 1: Reads data A from Accelerator Card 0, uses data A as the new current data, and saves it to the locally allocated location; Accelerator Card 2: Reads data B from Accelerator Card 1, uses data B as the new current data, and saves it to the locally allocated location; Accelerator Card 3: Reads data C from Accelerator Card 2, uses data C as the new current data, and saves it to the locally allocated location.

[0099] Begin the second round of reading and writing, such as Figure 6 As shown in (b): Accelerator Card 0: Reads data C from Accelerator Card 3, uses data C as the new current data, and saves it to the locally allocated location; Accelerator Card 1: Reads data D from Accelerator Card 0, uses data D as the new current data, and saves it to the locally allocated location; Accelerator Card 2: Reads data A from Accelerator Card 1, uses data A as the new current data, and saves it to the locally allocated location; Accelerator Card 3: Reads data B from Accelerator Card 2, uses data B as the new current data, and saves it to the locally allocated location.

[0100] Begin the third round of reading and writing, such as Figure 6 As shown in (c): Accelerator Card 0: Reads data B from Accelerator Card 3, uses data B as the new current data, and saves it to the locally allocated location; Accelerator Card 1: Reads data C from Accelerator Card 0, uses data C as the new current data, and saves it to the locally allocated location; Accelerator Card 2: Reads data D from Accelerator Card 1, uses data D as the new current data, and saves it to the locally allocated location; Accelerator Card 3: Reads data A from Accelerator Card 2, uses data A as the new current data, and saves it to the locally allocated location.

[0101] At this point, the entire AllGather operation is complete. The data in the four accelerator cards is completely identical, and each card has a complete data block AD. The entire process took 3 clock cycles, and the AD is used as the current calculation result.

[0102] According to some embodiments, in step S3, when the current operator is a data synchronization operator, multiple built-in synchronization computing units of the accelerator card are called to complete the corresponding calculation and obtain the current calculation result. Specifically, this can be achieved through steps S311-S314.

[0103] In step S311, when the current operator is the second data synchronization operator, the second data stored by each of the multiple accelerator cards is obtained as the current data, and the memory space of the data from other accelerator cards is allocated based on the remaining memory space of each of the multiple accelerator cards to obtain the allocation result. The second data synchronization operator includes the AllReduce operator.

[0104] The second data synchronization operator includes a data synchronization operator for performing calculations and synchronizing data using a specified aggregation method. This application uses the AllReduce operator as an example of the second data synchronization operator for illustration.

[0105] Among them, the AllReduce operator allows multiple accelerator cards with the same shape to calculate a result tensor through a specified aggregation method (such as summation, product, maximum value, etc.), and all devices eventually hold this result. It is one of the core collective communication operators in distributed computing.

[0106] Each accelerator card completes its own calculation, and the calculation result is recorded as the second data. All data blocks storing the second data are READY (ready) and are recorded as the current data of each accelerator card.

[0107] At the same time, based on the remaining memory space of each of the multiple accelerator cards, space is allocated to store data blocks from other accelerator cards, and this allocation result is recorded.

[0108] In step S312, the built-in synchronization calculation units of multiple accelerator cards are invoked. According to the pre-configured accelerator card data reading relationship, the current data of the corresponding accelerator card is read. Based on the read current data and the second data stored by each of the multiple accelerator cards, the corresponding calculation of the second data synchronization operator is performed to obtain the second calculation result.

[0109] The built-in synchronous computing units of multiple accelerator cards are invoked, and multiple accelerator cards begin to synchronize and compute simultaneously.

[0110] During the specific synchronization and calculation process, for each accelerator card, the current data of the corresponding accelerator card is read according to the pre-configured accelerator card data reading relationship. The corresponding calculation of the second data synchronization operator is performed based on the read current data and the stored second data to obtain the second calculation result, thus completing the synchronization and calculation process.

[0111] Understandably, the data read relationships of the accelerator cards are configured during the process of splitting the target model and adding data synchronization operators. Furthermore, the accelerator card IDs are also assigned during this period.

[0112] According to the example implementation, the compiler determines the specific accelerator card to which the data is allocated during compilation and defines the synchronization process as the data reading relationship of the accelerator card.

[0113] In step S313, the current data of multiple accelerator cards are updated according to the second calculation result, and the second calculation result is stored in the corresponding location according to the allocation result.

[0114] The second calculation result of each accelerator card is used as the new current data, and the second calculation result is stored in the allocated location to complete the data writing process.

[0115] In step S314, steps S312-S313 are repeated until the data in multiple accelerator cards are completely consistent, and the second calculation result of the last iteration is output as the current calculation result.

[0116] Repeat the process of data synchronization, computation, and writing until the data in multiple accelerator cards is completely consistent, and output completely consistent data as the current computation result.

[0117] At this point, the entire AllReduce process is complete, with all Tensor computation and synchronization performed internally within the hardware, without the need for the involvement of the upper-layer inference platform. The traditional method, however, involves the inference service reading data from multiple accelerator cards into memory, performing computation, and then writing it back to the accelerator cards. Because the inference service is required for synchronization and computation, the latency of this entire process is significantly higher than that of a purely hardware-based workflow.

[0118] To provide a more detailed description of the embodiments of this application, a specific embodiment is given. (Refer to...) Figure 7 In this embodiment, four acceleration cards are included, namely: Acceleration Card 0, Acceleration Card 1, Acceleration Card 2, and Acceleration Card 3, and the aggregation method is specified as summation calculation.

[0119] Each accelerator card completes its own calculation, and its corresponding second data is ready: the second data stored in accelerator card 0 is data A0, the second data stored in accelerator card 1 is data B0, the second data stored in accelerator card 2 is data C0, and the second data stored in accelerator card 3 is data D0. These data are used as the current data for each accelerator card, and each accelerator card is allocated space to store the other three data blocks.

[0120] The pre-configured data reading relationships of the accelerator cards include: accelerator card 0 reads data from accelerator card 3, accelerator card 1 reads data from accelerator card 0, accelerator card 2 reads data from accelerator card 1, and accelerator card 3 reads data from accelerator card 2.

[0121] The four accelerator cards began synchronizing and calculating, such as Figure 7 As shown in (a): Accelerator Card 0: Read D0 from Accelerator Card 3, add it to A0, save the result D0+A0 to the locally allocated location, and use D0+A0 as the new current data; Accelerator Card 1: Reads A0 from Accelerator Card 0, adds it to B0, saves the result A0+B0 to the locally allocated location, and uses A0+B0 as the new current data; Accelerator Card 2: Reads B0 from Accelerator Card 1, adds it to C0, saves the result B0+C0 to the locally allocated location, and uses B0+C0 as the new current data; Accelerator Card 3: Reads C0 from Accelerator Card 2, adds it to D0, saves the result C0+D0 to the locally allocated location, and uses C0+D0 as the new current data.

[0122] Begin the second round of synchronization and computation, such as Figure 7 As shown in (b): Accelerator Card 0: Read C0+D0 from Accelerator Card 3, add it to A0, save the result A0+C0+D0 to the locally allocated location, and use A0+C0+D0 as the new current data; Accelerator Card 1: Read D0+A0 from Accelerator Card 0, add it to B0, save the result A0+B0+D0 to the locally allocated location, and use A0+B0+D0 as the new current data; Accelerator Card 2: Reads A0+B0 from Accelerator Card 1, adds it to C0, saves the result A0+B0+C0 to the locally allocated location, and uses A0+B0+C0 as the new current data; Accelerator Card 3: Reads B0+C0 from Accelerator Card 2, adds it to D0, saves the result B0+C0+D0 to the locally allocated location, and uses B0+C0+D0 as the new current data.

[0123] Begin the third round of synchronization and computation, such as Figure 7 As shown in (c): Accelerator Card 0: Reads B0+C0+D0 from Accelerator Card 3, adds it to A0, saves the result A0+B0+C0+D0 to the locally allocated location, and uses A0+B0+C0+D0 as the new current data; Accelerator Card 1: Reads A0+C0+D0 from Accelerator Card 0, adds it to B0, saves the result A0+B0+C0+D0 to the locally allocated location, and uses A0+B0+C0+D0 as the new current data; Accelerator Card 2: Reads A0+B0+D0 from Accelerator Card 1, adds it to C0, saves the result A0+B0+C0+D0 to the locally allocated location, and uses A0+B0+C0+D0 as the new current data; Accelerator Card 3: Reads A0+B0+C0 from Accelerator Card 2, adds it to D0, saves the result A0+B0+C0+D0 to the locally allocated location, and uses A0+B0+C0+D0 as the new current data.

[0124] At this point, the entire AllReduce operation is complete. The data on the four accelerator cards is completely identical, each possessing a complete A0+B0+C0+D0. The entire process took three clock cycles, and A0+B0+C0+D0 is now used as the calculation result.

[0125] According to some embodiments, in step S3, when the current operator is a data synchronization operator, multiple built-in synchronization computing units of the accelerator card are called to complete the corresponding calculation and obtain the current calculation result. Specifically, this can be achieved through steps S321-S329.

[0126] In step S321, when the current operator is the third data synchronization operator, the third data stored by each of the multiple accelerator cards is divided into a first number of data blocks, and the data blocks are numbered. The first number is the number of multiple accelerator cards, and the third data synchronization operator includes the AllReduce + Scatter operator.

[0127] The third data synchronization operator includes a data synchronization operator used to perform calculations on tensors from multiple devices using a specified aggregation method, then split the aggregation result into sub-tensors and distribute them back to each device. This application uses the AllReduce + Scatter operator as an example of the third data synchronization operator for illustration.

[0128] Scatter is a fundamental one-way collective communication operator in distributed computing. Its core function is to split a complete tensor into multiple sub-tensors along a specified dimension by a designated "root device," and then distribute them to all devices in the cluster (including the root device itself). Each device only receives the corresponding split sub-tensor. The AllReduce + Scatter operator first performs aggregation operations (such as summation) on the tensors of multiple devices, and then splits the aggregation result into sub-tensors and distributes them back to each device.

[0129] Each accelerator card completes its own calculation, and the calculation result is recorded as the third data. All data blocks storing the third data are READY. The third data is then divided into a first number of data blocks. Here, the first number is the number of accelerator cards.

[0130] At the same time, the first number of data blocks are numbered.

[0131] In step S322, the data block with the first target number stored in each of the multiple accelerator cards is used as the current data, where the first target number is the number of the multiple accelerator cards.

[0132] For each accelerator card, the data block with the same number as the accelerator card number is used as the current data.

[0133] In step S323, the synchronous computing units built into multiple accelerator cards are invoked, and the current data in the corresponding accelerator card is read according to the pre-configured accelerator card data reading relationship, which is used as the second data to be processed.

[0134] The synchronous computing units built into multiple accelerator cards are invoked, and the multiple accelerator cards begin to synchronize data and start computing synchronously.

[0135] During the specific synchronization and calculation process, for each accelerator card, the current data of the corresponding accelerator card is read according to the pre-configured accelerator card data reading relationship, and recorded as the second data to be processed.

[0136] Understandably, the data read relationships of the accelerator cards are configured during the process of splitting the target model and adding data synchronization operators. Furthermore, the accelerator card IDs are also assigned during this period.

[0137] According to the example implementation, the compiler determines the specific accelerator card to which the data is allocated during compilation and defines the synchronization process as the data reading relationship of the accelerator card.

[0138] In step S324, data blocks with the same number as the current data from multiple accelerator cards are obtained as the first data to be processed.

[0139] For each accelerator card, obtain the data block with the same number as the current data and use it as the first data to be processed.

[0140] In step S325, based on the first data to be processed and the second data to be processed, the corresponding calculation of the third data synchronization operator is performed to obtain the third calculation result, and based on the third calculation result, the current data of multiple accelerator cards is updated.

[0141] For each accelerator card, the corresponding calculation of the third data synchronization operator is performed based on the first and second data to be processed to obtain the third calculation result, thus completing the synchronization and calculation process.

[0142] For each accelerator card, the obtained third calculation result will be used as the new current data.

[0143] In step S326, the third calculation result is stored in the original storage location of the first data to be processed, and steps S322-S326 are repeated until the third calculation result meets the preset conditions, wherein the preset conditions include completing the full accumulation of data blocks with the same number in multiple accelerator cards.

[0144] The third calculation result is stored in the original storage location of the first data to be processed, thus completing the data writing process.

[0145] The process of repeating data synchronization, calculation, and writing continues until the third calculation result meets the preset conditions.

[0146] The preset conditions include completing the full accumulation of data blocks with the same address number across multiple accelerator cards. For example, there are four accelerator cards: Accelerator Card 0, Accelerator Card 1, Accelerator Card 2, and Accelerator Card 3, with the specified aggregation method being summation. Accelerator Card 0 stores data A0-A3 as its second data, Accelerator Card 1 stores data B0-B3 as its second data, Accelerator Card 2 stores data C0-C3 as its second data, and Accelerator Card 3 stores data D0-D3 as its second data. For Accelerator Card 0, in the current iteration, the third calculation result is A1+B1+C1+D1, completing the full accumulation of data block numbered 1.

[0147] In step S327, the synchronous computing units built into multiple accelerator cards are invoked, and the latest data that meets the preset conditions in the corresponding accelerator card is read according to the data reading relationship of the accelerator cards, and used as the third data to be processed.

[0148] Then, data synchronization is performed by calling the built-in synchronous computing units of multiple accelerator cards, and multiple accelerator cards begin reading and writing simultaneously.

[0149] During the actual read and write process, according to the pre-configured data reading relationship of the accelerator card, the latest data that meets the preset conditions of the corresponding accelerator card is read and recorded as the third data to be processed, thus completing the data reading process.

[0150] In step S328, the third data to be processed is stored in the target storage location of multiple accelerator cards, wherein the target storage location is the storage location of the data block with the same number as the third data to be processed in the memory space of multiple accelerator cards.

[0151] For each accelerator card, the third data to be processed is stored in the storage location of the data block with the same number as the third data to be processed, thus completing the data writing process.

[0152] In step S329, steps S327-S328 are repeated until the data in multiple accelerator cards are completely consistent, and the completely consistent data is output as the current calculation result.

[0153] Repeat the data read and write process until the data in multiple accelerator cards is completely consistent, and output the completely consistent data as the current calculation result.

[0154] At this point, the entire AllReduce + Scatter computation process is complete, with data synchronization entirely within the hardware, requiring no involvement from the upper-layer inference platform. The traditional method, however, involves the inference service reading data from the accelerator card into memory, performing computation, and then writing it back to the accelerator card.

[0155] To provide a more detailed description of the embodiments of this application, a specific embodiment is given. Refer to Figures 8(a)-8(f). In this embodiment, four accelerator cards are included, namely: accelerator card 0, accelerator card 1, accelerator card 2, and accelerator card 3, and the aggregation method is specified as summation calculation.

[0156] Each accelerator card completes its own calculation, and its corresponding third data is ready: the third data stored in accelerator card 0 is data A, the third data stored in accelerator card 1 is data B, the third data stored in accelerator card 2 is data C, and the third data stored in accelerator card 3 is data D.

[0157] The third data is divided into four parts, which are used as four data blocks for each accelerator card and numbered as follows: data blocks for accelerator card 0 are A0-A3, data blocks for accelerator card 1 are B0-B3, data blocks for accelerator card 2 are C0-C3, and data blocks for accelerator card 3 are D0-D3.

[0158] The data block with the same number as the accelerator card is used as the current data of the current accelerator card: the current data of accelerator card 0 is A0, the current data of accelerator card 1 is B1, the current data of accelerator card 2 is C2, and the current data of accelerator card 3 is D3.

[0159] The pre-configured data reading relationships of the accelerator cards include: accelerator card 0 reads data from accelerator card 3, accelerator card 1 reads data from accelerator card 0, accelerator card 2 reads data from accelerator card 1, and accelerator card 3 reads data from accelerator card 2.

[0160] The four accelerator cards begin synchronization and calculation, as shown in Figure 8(a): Accelerator Card 0: Reads the current data D3 from Accelerator Card 3, adds it to A3 (the first data to be processed) to get A3+D3, and uses A3+D3 as the new current data of Accelerator Card 0, and writes A3+D3 to the original position of A3; Accelerator Card 1: Reads the current data A0 from Accelerator Card 0, adds it to B0 (the first data to be processed) to get A0+B0, and uses A0+B0 as the new current data of Accelerator Card 1, and writes A0+B0 to the original position of B0. Accelerator Card 2: Reads the current data B1 from Accelerator Card 1, adds it to C1 (the first data to be processed) to get B1+C1, and uses B1+C1 as the new current data of Accelerator Card 2, and writes B1+C1 to the original position of C1. Accelerator Card 3: Reads the current data C2 from Accelerator Card 2, adds it to D2 (the first data to be processed) to get C2+D2, and uses C2+D2 as the new current data of Accelerator Card 3, and writes C2+D2 to the original position of D2.

[0161] The second round of synchronization and computation begins, as shown in Figure 8(b): Accelerator Card 0: Reads the current data C2+D2 from Accelerator Card 3, adds it to A2 (the first data to be processed) to get A2+C2+D2, and uses A2+C2+D2 as the new current data of Accelerator Card 0, and writes the result to the original position of A2; Accelerator Card 1: Reads the current data A3+D3 from Accelerator Card 0, adds it to B3 (the first data to be processed) to get A3+B3+D3, and uses A3+B3+D3 as the new current data of Accelerator Card 1, and writes the result to the original position B2. Accelerator Card 2: Reads the current data A0+B0 from Accelerator Card 1, adds it to C0 (the first data to be processed) to get A0+B0+C0, and uses A0+B0+C0 as the new current data of Accelerator Card 2, and writes the result to the original position of C0. Accelerator Card 3: Reads the current data B1+C1 from Accelerator Card 2, adds it to D1 (the first data to be processed) to get B1+C1+D1, and uses B1+C1+D1 as the new current data of Accelerator Card 3, and writes the result to the original position of D1.

[0162] The third round of synchronization and computation begins, as shown in Figure 8(c): Accelerator Card 0: Reads the current data B1+C1+D1 from Accelerator Card 3, adds it to A1 (the first data to be processed) to get A1+B1+C1+D1, and uses A1+B1+C1+D1 as the new current data of Accelerator Card 0. Writes the result back to the original position of A1, and completes the full accumulation of the data block numbered 1. Accelerator Card 1: Reads the current data A2+C2+D2 from Accelerator Card 0, adds it to B2 (the first data to be processed) to get A2+B2+C2+D2, and uses A2+B2+C2+D2 as the new current data of Accelerator Card 1. Writes the result to the original position of B2, and completes the full accumulation of the data block numbered 2. Accelerator Card 2: Reads the current data A3+B3+D3 from Accelerator Card 1, adds it to C3 (the first data to be processed) to get A3+B3+C3+D3, and uses A3+B3+C3+D3 as the new current data of Accelerator Card 2. Writes the result to the original position of C3, and completes the full accumulation of the data block numbered 3. Accelerator Card 3: Reads the current data A0+B0+C0 from Accelerator Card 2, adds it to D0 (the first data to be processed) to get A0+B0+C0+D0, and uses A0+B0+C0+D0 as the new current data of Accelerator Card 3. Writes the result to the original position of D0, and completes the full accumulation of the data block numbered 0.

[0163] The first round of synchronization begins, as shown in Figure 8(d): Accelerator Card 0: Reads the latest full data A0+B0+C0+D0 that has already been accumulated with the same address number from Accelerator Card 3, and writes the result back to the original position A0; Accelerator Card 1: Reads the latest full data A1+B1+C1+D1 that has already been accumulated with the same address number from Accelerator Card 0, and writes the result back to the original B1 position; Accelerator Card 2: Reads the latest full data A2+B2+C2+D2 that has already been accumulated with the same address number from Accelerator Card 1, and writes the result back to the original position of C2; Accelerator Card 3: Reads the latest full data A3+B3+C3+D3 that has already been accumulated with the same address number from Accelerator Card 2, and writes the result back to the original position D3; The second round of synchronization begins, as shown in Figure 8(e): Accelerator Card 0: Read the latest full data A3+B3+C3+D3 that has already been accumulated with the same address number from Accelerator Card 3, and write the result back to the original position A3; Accelerator Card 1: Reads the latest full data A0+B0+C0+D0 that has already been accumulated with the same address number from Accelerator Card 0, and writes the result back to the original B0 position; Accelerator Card 2: Reads the latest full data A1+B1+C1+D1 that has already been accumulated with the same address number from Accelerator Card 1, and writes the result back to the original position of C1; Accelerator Card 3: Reads the latest full data A2+B2+C2+D2 that has already been accumulated with the same address number from Accelerator Card 2, and writes the result back to the original position D2; The third round of synchronization begins, as shown in Figure 8(f): Accelerator Card 0: Read the latest full data A2+B2+C2+D2 that has already been accumulated with the same address number from Accelerator Card 3, and write the result back to the original position A2; Accelerator Card 1: Reads the latest full data A3+B3+C3+D3 that has already been accumulated with the same address number from Accelerator Card 0, and writes the result back to the original B3 position; Accelerator Card 2: Reads the latest full data A0+B0+C0+D0 that has already been accumulated with the same address number from Accelerator Card 1, and writes the result back to the original position of C0; Accelerator Card 3: Reads the latest full data A2+B2+C2+D2 from Accelerator Card 2, which has already been accumulated with the same address number, and writes the result back to the original position D2.

[0164] At this point, the entire AllReduce + Scatter computation process is complete. The Tensors in the four accelerator cards are completely identical, each possessing the complete A0+B0+C0+D0 - A3+B3+C3+D3. The process takes a total of 6 clock cycles, and A0+B0+C0+D0 - A3+B3+C3+D3 is taken as the current computation result.

[0165] According to experiments, compared to the synchronization latency of over 1ms in traditional synchronization methods, the synchronization latency of the task processing method provided in this application is reduced to less than 100µs. This application adopts a hardware-software co-operated communication operator, achieving full hardware synchronization, reducing scheduling overhead, and integrating data synchronization and data computation simultaneously, thus saving synchronization time.

[0166] The following describes an apparatus embodiment of this application, which can be used to perform the method embodiment of this application. For details not disclosed in the apparatus embodiment of this application, please refer to the method embodiment of this application.

[0167] Figure 9 A block diagram of a task processing apparatus according to an exemplary embodiment is shown.

[0168] Figure 9 The apparatus shown can perform the processing method of the task according to the embodiments of this application.

[0169] like Figure 9As shown, the task processing device may include: See Figure 9 Referring to the preceding description, the splitting module 910 is used to split the target model and insert data synchronization operators to obtain multiple model blocks.

[0170] The loading module 920 is used to load multiple model blocks into multiple accelerator cards, wherein the multiple accelerator cards are interconnected in pairs.

[0171] The computation module 930 is used to respond to the acquired inference task by performing parallel computation on the inference task based on multiple accelerator cards to obtain the inference result. The computation includes the corresponding computation of model operators in the target model and the corresponding computation of data synchronization operators.

[0172] The device performs functions similar to those described above; other functions are described in the preceding descriptions and will not be repeated here.

[0173] This application discloses an electronic device, including: a processor; and a memory storing a computer program, which, when executed by the processor, causes the processor to execute the above-described instruction generation method.

[0174] For example, refer to Figure 10 , Figure 10 The illustrated electronic device 1000 includes a processor 1001 and a memory 1003. The processor 1001 and the memory 1003 are connected, for example, via a bus 1002. Optionally, the electronic device 1000 may also include a transceiver 1004. It should be noted that in practical applications, the transceiver 1004 is not limited to one type, and the structure of this electronic device 1000 does not constitute a limitation on the embodiments of this application.

[0175] Processor 1001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in this application. Processor 1001 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0176] Bus 1002 may include a pathway for transmitting information between the aforementioned components. Bus 1002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 1002 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 10 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0177] The memory 1003 may be a ROM (Read Only Memory) or other type of static storage device capable of storing static information and instructions, RAM (Random Access Memory) or other type of dynamic storage device capable of storing information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other storage medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0178] The memory 1003 is used to store application code that executes the solution of this application, and its execution is controlled by the processor 1001. The processor 1001 is used to execute the application code stored in the memory 1003 to implement the content shown in the foregoing method embodiments.

[0179] Figure 10 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0180] This application discloses a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, causes the processor to execute an instruction generation method.

[0181] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0182] The above are only some embodiments of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for processing a task, characterized in that, include: The target model is split and a data synchronization operator is inserted to obtain multiple model blocks; The multiple model blocks are loaded into multiple accelerator cards, wherein the multiple accelerator cards are interconnected in pairs; In response to the acquired inference task, the inference task is computed in parallel based on the multiple accelerator cards to obtain the inference result, wherein the computation includes the corresponding computation of the model operator in the target model and the corresponding computation of the data synchronization operator.

2. The method according to claim 1, characterized in that, The target model is split and a data synchronization operator is inserted to obtain multiple model blocks, including: The preset weight parameters of the target model are split into tensors in parallel to obtain the splitting results; Extract the computing nodes that need data synchronization from the splitting results, and insert the corresponding data synchronization operators at the positions of the computing nodes to obtain the multiple model blocks.

3. The method according to claim 1, characterized in that, Any two of the multiple accelerator cards are connected via a hardware interface.

4. The method according to claim 2, characterized in that, In response to the acquired inference task, the inference task is computed in parallel using the multiple accelerator cards to obtain the inference result, including: In response to the acquired task to be inferred, the task to be inferred is written into the memory of the plurality of accelerator cards; Based on the multiple accelerator cards, the model operators and / or data synchronization operators in the corresponding model blocks are executed in parallel, and data exchange is performed when the data synchronization operator is executed, until the operators in the corresponding model blocks are traversed, and the calculation result is output as the inference result.

5. The method according to claim 4, characterized in that, Based on the multiple accelerator cards, model operators and / or data synchronization operators in the corresponding model blocks are executed in parallel, and data exchange is performed when the data synchronization operator is executed, until the operators in the corresponding model blocks are traversed, and the calculation results are output as the inference results, including: S1: Based on the multiple accelerator cards, extract the current operator from the operators in the corresponding model block; S2: When the current operator is a model operator, the computing units built into the multiple accelerator cards are called to complete the corresponding calculation and obtain the current calculation result; S3: When the current operator is the data synchronization operator, the synchronous calculation unit built into the multiple accelerator cards is called to complete the corresponding calculation and obtain the current calculation result; S4: Repeat S1-S3 until all the accelerator cards have traversed the operators in the corresponding model blocks, and output the current calculation result of the last iteration as the inference result.

6. The method according to claim 5, characterized in that, Step S3 includes: S301: When the current operator is the first data synchronization operator, the first data stored by each of the multiple accelerator cards is obtained as the current data, and the memory space of the data from other accelerator cards is allocated based on the remaining memory space of each of the multiple accelerator cards to obtain the allocation result, wherein the first data synchronization operator includes the AllGather operator; S302: Call the synchronous calculation unit built into the multiple accelerator cards, and read the current data of the corresponding accelerator card according to the pre-configured accelerator card data reading relationship, as the first calculation result of the multiple accelerator cards; S303: Based on the first calculation result, update the current data of the plurality of accelerator cards, and store the first calculation result in the corresponding location according to the allocation result; S304: Repeat steps S302-S303 until the data in the multiple accelerator cards are completely consistent, and output the completely consistent data as the current calculation result.

7. The method according to claim 5, characterized in that, Step S3 includes: S311: When the current operator is the second data synchronization operator, the second data stored by each of the multiple accelerator cards is obtained as the current data, and the memory space of the data from other accelerator cards is allocated based on the remaining memory space of each of the multiple accelerator cards to obtain the allocation result, wherein the second data synchronization operator includes the AllReduce operator; S312: Call the built-in synchronization calculation unit of the multiple accelerator cards, read the current data of the corresponding accelerator card according to the pre-configured accelerator card data reading relationship, and perform the corresponding calculation of the second data synchronization operator according to the read current data and the second data stored in each of the multiple accelerator cards to obtain the second calculation result; S313: Based on the second calculation result, update the current data of the plurality of accelerator cards, and store the second calculation result in the corresponding location according to the allocation result; S314: Repeat steps S312-S313 until the data in the multiple accelerator cards are completely consistent, and output the second calculation result of the last iteration as the current calculation result.

8. The method according to claim 5, characterized in that, Step S3 includes: S321: When the current operator is a third data synchronization operator, the third data stored by each of the multiple accelerator cards is divided into a first number of data blocks, and the data blocks are numbered, wherein the first number is the number of the multiple accelerator cards, and the third data synchronization operator includes the AllReduce + Scatter operator. S322: Take the data block with the first target number stored in each of the plurality of accelerator cards as the current data, wherein the first target number is the number of the plurality of accelerator cards; S323: Call the synchronous computing unit built into the multiple accelerator cards, and read the current data in the corresponding accelerator card according to the pre-configured accelerator card data reading relationship, as the second data to be processed; S324: Obtain the data block with the same number as the current data from the plurality of accelerator cards, and use it as the first data to be processed; S325: Based on the first data to be processed and the second data to be processed, perform the corresponding calculation of the third data synchronization operator to obtain the third calculation result, and update the current data of the plurality of accelerator cards based on the third calculation result; S326: Store the third calculation result in the original storage location of the first data to be processed, and repeat steps S322-S326 until the third calculation result meets the preset conditions, wherein the preset conditions include completing the full accumulation of data blocks with the same number in the multiple accelerator cards; S327: Call the synchronous computing unit built into the multiple accelerator cards, and read the latest data that meets the preset conditions in the corresponding accelerator card according to the data reading relationship of the accelerator cards, as the third data to be processed; S328: The third data to be processed is stored in the target storage location of the plurality of accelerator cards, wherein the target storage location is the storage location of the data block with the same number as the third data to be processed in the memory space of the plurality of accelerator cards; S329: Repeat steps S327-S328 until the data in the multiple accelerator cards are completely consistent, and output the completely consistent data as the current calculation result.

9. A task processing apparatus, characterized in that, include: The splitting module is used to split the target model and insert data synchronization operators to obtain multiple model blocks; A loading module is used to load the plurality of model blocks into a plurality of accelerator cards, wherein the plurality of accelerator cards are interconnected in pairs; The computing module is used to respond to the acquired inference task by performing parallel computing on the inference task based on the multiple accelerator cards to obtain the inference result. The computing includes the corresponding computing of the model operators in the target model and the corresponding computing of the data synchronization operators.

10. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-8.

11. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they implement the method as described in any one of claims 1-8.

Images