Data flow policy generation method and apparatus, electronic device, and storage medium

Abstract

Embodiments of the present application provide a data flow strategy generation method and device, electronic equipment and storage medium. The data flow strategy generation method comprises: obtaining a computation graph corresponding to a data processing task; generating an inter-stage data flow strategy according to the computation graph and an execution cost, wherein the inter-stage data flow strategy comprises a strategy of dividing operators included in the computation graph into a plurality of pipeline stages, and each pipeline stage comprises at least one operator; generating a plurality of intra-stage data flow strategies corresponding to the plurality of pipeline stages according to the inter-stage data flow strategy; updating the execution cost according to the plurality of intra-stage data flow strategies to optimize the inter-stage data flow strategy, and obtaining a target inter-stage data flow strategy for executing the data processing task and a plurality of corresponding target intra-stage data flow strategies. The data flow strategy generation scheme provided by the present scheme can be applied to different data processing tasks and hardware, and has strong applicability.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a data flow strategy generation method, apparatus, electronic device, and storage medium. Background Technology

[0002] Data-flow architecture is a novel computing architecture that uses a multi-core architecture for spatial and temporal computation. Compared to shared-memory architectures such as CPUs (Central Processing Units) and GPUs (Graphics Processing Units), data-flow architecture explicitly divides tasks into microtasks and maps them to multiple processing elements (PEs), and explicitly orchestrates data movement between processing elements. In some scenarios, data-flow architecture can achieve higher system utilization, energy efficiency, and model capacity than shared-memory architecture. Therefore, data-flow architecture can be applied to perform deep learning tasks, such as training and inference of neural network models.

[0003] When optimizing a dataflow architecture, a dataflow strategy needs to be built to map the computation graph to the hardware. Currently, when optimizing a dataflow architecture, the dataflow strategy is built manually, the computation graph is manually partitioned into multiple operators, and the partitioned operators are mapped to specific hardware.

[0004] However, optimizing the data flow architecture by manually constructing data flow strategies requires manual operation and professional knowledge, which cannot be extended to different data processing tasks and hardware, thus resulting in poor applicability. Summary of the Invention

[0005] In view of this, embodiments of this application provide a data flow strategy generation method, apparatus, electronic device, and storage medium to at least solve or alleviate the above-mentioned problems.

[0006] According to a first aspect of the present application, a data flow strategy generation method is provided, comprising: acquiring a computation graph corresponding to a data processing task; generating an inter-stage data flow strategy based on the computation graph and an execution cost, wherein the inter-stage data flow strategy includes a strategy of dividing operators included in the computation graph into multiple pipeline stages, and each pipeline stage includes at least one operator; generating multiple intra-stage data flow strategies corresponding to the multiple pipeline stages based on the inter-stage data flow strategy; updating the execution cost based on the multiple intra-stage data flow strategies to optimize the inter-stage data flow strategy, thereby obtaining a target inter-stage data flow strategy for executing the data processing task and corresponding multiple target intra-stage data flow strategies.

[0007] According to a second aspect of the embodiments of this application, a data flow strategy generation apparatus is provided, comprising: an acquisition unit for acquiring a computation graph corresponding to a data processing task; a first generation unit for generating an inter-stage data flow strategy based on the computation graph and an execution cost, wherein the inter-stage data flow strategy includes a strategy of dividing operators included in the computation graph into multiple pipeline stages, and each pipeline stage includes at least one operator; a second generation unit for generating multiple intra-stage data flow strategies corresponding to the multiple pipeline stages based on the inter-stage data flow strategy; and an update unit for updating the execution cost based on the multiple intra-stage data flow strategies to optimize the inter-stage data flow strategy, thereby obtaining a target inter-stage data flow strategy for executing the data processing task and corresponding multiple target intra-stage data flow strategies.

[0008] According to a third aspect of the present application, an electronic device is provided, including: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other through the communication bus; the memory is used to store at least one executable instruction, which causes the processor to perform the operation corresponding to the method described in the first aspect.

[0009] According to a fourth aspect of the embodiments of this application, a computer storage medium is provided, on which a computer program is stored, the program being executed by a processor according to the method described in the first aspect above.

[0010] According to a fifth aspect of the embodiments of this application, a computer program product is provided, including computer instructions that instruct a computing device to perform the method of the first aspect described above.

[0011] According to the data flow strategy generation scheme provided in the embodiments of this application, the data flow strategy includes inter-stage data flow strategy and multiple intra-stage data flow strategies. The inter-stage data flow strategy can be determined based on the computation graph and execution cost. The intra-stage data flow strategy is determined based on the inter-stage data flow strategy. The execution cost can be updated based on the intra-stage data flow strategy to optimize the inter-stage data flow strategy. Based on the optimized inter-stage data flow strategy, a new intra-stage data flow strategy can be generated. Repeating the above optimization process can obtain a data flow strategy for data processing tasks. This data flow strategy includes a target inter-stage data flow strategy and multiple corresponding target intra-stage data flow strategies, thereby realizing the automated generation of data flow strategies. It can be applied to different data processing tasks and hardware resources and has strong applicability. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0013] Figure 1 This is a schematic diagram of an exemplary system applied in one embodiment of this application;

[0014] Figure 2 This is a flowchart of a data flow strategy generation method according to an embodiment of this application;

[0015] Figure 3 This is a schematic diagram of pipeline parallelism according to one embodiment of this application;

[0016] Figure 4 This is a schematic diagram of data parallelism in one embodiment of this application;

[0017] Figure 5 This is a schematic diagram of model parallelism in one embodiment of this application;

[0018] Figure 6 This is a schematic diagram of hardware resources according to an embodiment of this application;

[0019] Figure 7 This is a flowchart of a method for generating a data flow strategy within a stage according to an embodiment of this application;

[0020] Figure 8 This is a schematic diagram of the data flow strategy generation process according to an embodiment of this application;

[0021] Figure 9 This is a schematic diagram of the data flow strategy generation process according to another embodiment of this application;

[0022] Figure 10 This is a schematic diagram of a data stream generation apparatus according to an embodiment of this application;

[0023] Figure 11 This is a schematic diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0024] The present application is described below based on embodiments, but it is not limited to these embodiments. In the detailed description of the present application below, certain specific details are described in detail. Those skilled in the art can fully understand the present application without these details. To avoid obscuring the essence of the present application, well-known methods, processes, and flows are not described in detail. Furthermore, the accompanying drawings are not necessarily drawn to scale.

[0025] First, some nouns or terms that appear in the description of the embodiments of this application shall be interpreted as follows.

[0026] Data-Flow Architecture: Data-flow architecture is a computing architecture that differs from shared memory architectures such as CPU and GPU. It uses a multi-core architecture for spatial and temporal computation, divides data and tasks into microtasks and maps them to multiple processing units, and orchestrates the data movement between processing units.

[0027] Processing Element (PE): A processing element is a basic computational block in a dataflow architecture. Processing elements are connected via an on-chip network.

[0028] Network on Chip (NoC): Network on Chip (NoC) is a novel communication method for System on Chip (SoC). NoC connects multiple nodes on a chip, enabling reliable communication between them. The topologies that can be formed by the nodes in a NoC include 2D / 3D mesh networks, torus networks, and ring networks.

[0029] Computational graph: A computational graph is a directed graph used to represent computational flows. It includes nodes and edges connecting nodes. Nodes represent operators (OPs) in the computational flow, and edges connecting nodes indicate that there is a data dependency or control dependency between the two connected nodes.

[0030] Tensor: A tensor is data that is passed between processing units. Tensors are multi-dimensional data, such as matrices and vectors.

[0031] Data Parallelism: Data parallelism is a common form of distributed training of deep learning models. It involves dividing the training data samples into multiple parts, replicating the same model parameters on different devices and training them simultaneously, and then aggregating the parameter gradients generated during training through the network to update the parameters.

[0032] Model Parallelism: Model parallelism can divide the entire computation graph into different devices according to different stages, or divide a single operator (data) into multiple devices for joint computation and data exchange or aggregation at specific locations.

[0033] Data flow strategy: The data flow strategy refers to a general strategy for partitioning the computation graph, dividing operators into micro-operators, splitting tensors into micro-tensors, and mapping micro-operators to processing units.

[0034] Application Environment of this Application

[0035] This application proposes a data flow policy generation scheme. The entire data flow policy generation scheme is relatively universal and can be used on various hardware devices employing a data flow architecture, such as data centers, servers, personal computers, Internet of Things (IoT) devices, and embedded devices. The data flow policy generation scheme is independent of the hardware on which the computing device executing the scheme is deployed.

[0036] Exemplary System

[0037] Figure 1 An exemplary system for a data flow policy generation method applicable to embodiments of this application is shown. For example... Figure 1 As shown, the system may include a cloud server 102, a communication network 104, and at least one user device 106. Figure 1 The example shown is multiple user devices 106. It should be noted that the solution in this embodiment can be applied to both the cloud server 102 and the user devices 106.

[0038] The cloud server 102 can be any suitable device for storing information, data, programs, and / or any other suitable type of content, including but not limited to distributed storage system devices, server clusters, computing cloud server clusters, etc. In some embodiments, the cloud server 102 can perform any suitable function. For example, in some embodiments, the cloud server 102 can be used to generate a data flow policy when performing data processing tasks through a data flow architecture. As an optional example, in some embodiments, the cloud server 102 can generate a data flow policy for a data processing task and execute the data processing task based on the data flow policy. In other embodiments, the cloud server 102 can generate a data flow policy for a data processing task and send the generated data flow policy to the user device 106, which then executes the data processing task according to the data flow policy.

[0039] Communication network 104 can be any suitable combination of one or more wired and / or wireless networks. For example, communication network 104 can include any one or more of the following: the Internet, intranet, wide area network (WAN), local area network (LAN), wireless network, digital subscriber line (DSL) network, frame relay network, asynchronous transfer mode (ATM) network, virtual private network (VPN), and / or any other suitable communication network. User equipment 106 can be connected to communication network 104 via one or more communication links (e.g., communication link 112), and communication network 104 can be linked to cloud server 102 via one or more communication links (e.g., communication link 114). Communication links can be any communication link suitable for transmitting data between cloud server 102 and user equipment 106, such as network links, dial-up links, wireless links, hardwired links, any other suitable communication links, or any suitable combination of such links.

[0040] User device 106 may include any one or more user devices suitable for interaction. In some embodiments, when a data flow policy is generated by cloud server 102, user device 106 may send a policy generation request to cloud server 102, the policy generation request including relevant information about the data processing task, to trigger cloud server 102 to generate a data flow policy according to the request and feed the generated data flow policy back to user device 106. User device 106 may include any suitable type of device; for example, user device 106 may include mobile devices, tablet computers, laptop computers, desktop computers, wearable computers, vehicle systems, and / or any other suitable type of user device.

[0041] In another exemplary system applicable to the data flow policy generation method of this application embodiment, the system may include a host computer and one or more accelerator cards, with the host computer and accelerator cards connected via a communication network such as a bus. The host computer can generate a data flow policy for executing data processing tasks through a data flow architecture and schedule the accelerator cards to execute data processing tasks according to the data flow policy. The accelerator card includes a network on-chip (NoC) composed of multiple processing units (PEs). Processing units (PEs) within the same network on-chip (NoC) communicate through the network on-chip (NoC), and different accelerator cards communicate with each other via Ethernet or the like. The host computer can deploy the data flow policy onto one accelerator card or onto multiple accelerator cards; this is not limited in this embodiment.

[0042] This application focuses on the process of generating data flow strategies by the cloud server 102 or the host computer. The generation process of data flow strategies will be described in detail later.

[0043] Data Stream Strategy Generation Method

[0044] Based on the above system, this application provides a data flow strategy generation method, which will be described in detail below through multiple embodiments.

[0045] Figure 2 This is a flowchart of a data flow strategy generation method according to an embodiment of this application. Figure 2 As shown, the data flow strategy generation method includes the following steps:

[0046] Step 201: Obtain the computation graph corresponding to the data processing task.

[0047] A computation graph can represent the direction of data flow when performing data processing tasks. It includes multiple operators represented by nodes, and the connections between nodes indicate data dependencies or control dependencies between operators. Data dependencies mean that subsequent operators need to perform operations based on the output data of preceding operators, while control dependencies mean that subsequent operators need to perform operations after the output data of preceding operators are available. Operators are the various operations that need to be performed when executing data processing tasks, such as matrix multiplication and matrix addition.

[0048] Data processing tasks can be either training tasks for neural network models or inference tasks for neural network models.

[0049] Step 202: Generate inter-stage data flow strategy based on the computation graph and execution cost.

[0050] Execution cost refers to the cost of performing a data processing task based on a corresponding data flow strategy. This cost could include, for example, the hardware resources or time required to perform the task. Since generating a data flow strategy is an optimization process, the execution cost in the first round of generating inter-stage data flow strategies can be a preset value that has little or no impact on the generated strategy. From the second round onwards, the execution cost can be determined based on the data flow strategy generated in the previous round. During the optimization process of the data flow strategy, the execution cost can indicate whether the hardware resource consumption and / or time consumption requirements are met when performing data processing tasks based on the generated data flow strategy.

[0051] To improve data processing efficiency, operator parallelism can be achieved through various methods such as pipeline parallelism, data parallelism, and model parallelism. Pipeline parallelism divides the computation graph into multiple pipeline stages based on data flow. Each pipeline stage includes one or more operators, and the output data of the preceding pipeline stage serves as the input data for the subsequent pipeline stage; the data processing proceeds in a pipelined manner. Data parallelism splits the data into input data for different operators, achieving data parallelism. Model parallelism parallelizes multiple operators that depend on the same operator, based on dependencies between operators.

[0052] Figure 3 This is a schematic diagram of pipeline parallelism according to one embodiment of this application. Figure 4 This is a schematic diagram of data parallelism in one embodiment of this application. Figure 5 This is a schematic diagram of a model parallelism according to an embodiment of this application.

[0053] like Figure 3 As shown, the computation graph includes OP0 to OP4. OP0 and OP1 are assigned to pipeline stage 0, OP2 to pipeline stage 1, OP3 to pipeline stage 2, and OP4 to pipeline stage 3. Different pipeline stages are mapped to different processing units on the hardware to achieve pipeline parallelism.

[0054] like Figure 4 As shown, the data to be processed is split into three input data, which are then used as inputs to OP0, OP1, and OP2 respectively. The output data of OP0, OP1, and OP2 are aggregated to obtain the data processing result corresponding to the data to be processed. OP0 and its corresponding input data are mapped to instance 0 on the hardware, OP1 and its corresponding input data are mapped to instance 1 on the hardware, and OP2 and its input data are mapped to instance 2 on the hardware. Instance 0, instance 1, and instance 2 are executed in parallel to achieve data parallelism.

[0055] like Figure 5 As shown, OP1, OP2, and OP3 have data or control dependencies with OP0. By mapping OP0, OP1, OP2, and OP3 onto hardware, OP1, OP2, and OP3 can be executed in parallel after OP0 outputs data, thus achieving model parallelism.

[0056] Inter-stage data flow strategies include strategies for dividing the operators in a computation graph into multiple pipeline stages. Specifically, based on these strategies, multiple operators in the computation graph can be divided into multiple pipeline stages, ensuring that each pipeline stage includes at least one operator. The data flow strategy includes inter-stage data flow strategies, which allow the computation graph to be divided into multiple pipeline stages, each of which can process data in a pipelined parallel manner.

[0057] Step 203: Based on the inter-stage data flow strategy, generate multiple intra-stage data flow strategies corresponding to multiple pipeline stages.

[0058] Since the inter-stage data flow strategy can determine the operators included in the pipeline stage, after generating the inter-stage data flow strategy, the data processing method within the pipeline stage can be determined based on the inter-stage data flow strategy, thereby generating the intra-stage data flow strategy that can indicate the implementation method of operators within the pipeline stage. Each pipeline stage corresponds to an intra-stage data flow strategy, and the intra-stage data flow strategy can indicate the implementation method of operators within the corresponding pipeline stage.

[0059] Step 204: Update the execution cost based on the data flow strategies within multiple stages to optimize the data flow strategies between stages, and obtain the target inter-stage data flow strategy and the corresponding multiple target intra-stage data flow strategies for performing data processing tasks.

[0060] After obtaining multiple intra-stage data flow strategies, the execution cost can be updated based on each intra-stage data flow strategy. Then, steps 202 and 203 above can be executed based on the updated execution cost to generate a new inter-stage data flow strategy. Furthermore, a new intra-stage data flow strategy is generated based on the new inter-stage data flow strategy, thereby optimizing the inter-stage data flow strategy. After optimizing the inter-stage data flow strategy and obtaining the target inter-stage data flow strategy, multiple intra-stage data flow strategies corresponding to the target inter-stage data flow strategy can be obtained. This results in a data flow strategy that includes the target inter-stage data flow strategy and the data flow strategies for each target intra-stage, which can be used to execute data processing tasks.

[0061] The intra-stage data flow strategy is determined based on the inter-stage data flow strategy, and the inter-stage data flow strategy is optimized based on execution cost. Therefore, by optimizing the inter-stage data flow strategy according to the execution cost, a target inter-stage data flow strategy that meets the requirements can be obtained. In turn, multiple target intra-stage data flow strategies corresponding to the target inter-stage data flow strategy can be obtained, so as to obtain a data flow strategy for the execution of data processing tasks, which includes the target inter-stage data flow strategy and the data flow strategies of each target intra-stage.

[0062] In this embodiment, the data flow strategy includes an inter-stage data flow strategy and multiple intra-stage data flow strategies. The inter-stage data flow strategy can be determined based on the computation graph and execution cost. The intra-stage data flow strategy is determined based on the inter-stage data flow strategy. The execution cost can be updated based on the intra-stage data flow strategy to optimize the inter-stage data flow strategy. A new intra-stage data flow strategy can be generated based on the optimized inter-stage data flow strategy. Repeating the above optimization process can obtain a data flow strategy for data processing tasks. This data flow strategy includes a target inter-stage data flow strategy and multiple corresponding target intra-stage data flow strategies, thereby realizing the automated generation of data flow strategies. It can be applied to different data processing tasks and hardware resources and has strong applicability.

[0063] Optimizing data flow strategies by breaking them down into inter-stage and intra-stage data flow strategies allows for the determination of intra-stage data flow strategies for multiple pipeline stages, reducing the search space for intra-stage data flow strategies and thus shortening the time required to generate data flow strategies, thereby improving the efficiency of data flow strategy generation.

[0064] In one possible implementation, when optimizing the inter-stage data flow strategy, the execution cost can be updated based on the current multiple intra-stage data flow strategies, and it can be determined whether the updated execution cost meets the optimization termination condition. If the updated execution cost does not meet the optimization termination condition, steps 202 and 203 are returned to the execution process based on the updated execution cost to optimize the current inter-stage data flow strategy. If the updated execution cost meets the optimization termination condition, the current inter-stage data flow strategy and the corresponding multiple intra-stage data flow strategies are determined as the target inter-stage data flow strategy and the corresponding multiple target intra-stage data flow strategies.

[0065] Execution cost can indicate whether the current inter-stage data flow strategy and the corresponding intra-stage data flow strategies meet the requirements of executing data processing tasks. Therefore, the optimization termination condition can be determined based on the requirements of executing data processing tasks, and then the optimization of the inter-stage data flow strategy can be determined based on the matching relationship between execution cost and optimization termination condition.

[0066] In this embodiment, after generating multiple corresponding intra-stage data flow strategies based on the inter-stage data flow strategy, the execution cost can be updated according to the intra-stage data flow strategy. Based on the matching relationship between the updated execution cost and the optimization termination condition, it is determined whether to continue optimizing the inter-stage data flow strategy. After optimization, a target inter-stage data flow strategy and multiple corresponding target intra-stage data flow strategies for data processing task execution are obtained. The optimization process of the inter-stage data flow strategy is controlled based on the matching relationship between the execution cost and the optimization termination condition. Simultaneously, the execution cost serves as the basis for optimizing the inter-stage data flow strategy, ensuring that the optimization process proceeds smoothly and that the target inter-stage data flow strategy can be obtained quickly, thus improving the efficiency of data flow strategy generation.

[0067] In one possible implementation, the optimization termination condition could be execution cost convergence, or the execution time of the data processing task being less than a duration threshold.

[0068] Execution cost indicates the cost of performing data processing tasks based on the corresponding inter-stage data flow strategy and the corresponding intra-stage data flow strategy. During the optimization of the inter-stage data flow strategy, the execution cost needs to be updated, and a new inter-stage data flow strategy is generated based on the updated execution cost. If the execution cost converges during the optimization process—that is, the execution cost no longer changes with changes in the intra-stage data flow strategy, or only changes very slightly—it indicates that the cost of performing data processing tasks based on the current inter-stage data flow strategy and the corresponding intra-stage data flow strategy has reached a relatively optimal state. Therefore, optimization of the inter-stage data flow strategy can be stopped, and execution cost convergence can be used as the termination condition for optimization.

[0069] The intra-stage data flow strategy is determined based on the inter-stage data flow strategy. By optimizing the inter-stage data flow strategy, more data processing processes within the pipeline stage can be performed in parallel, thereby reducing the execution time of data processing tasks. If, during the optimization of the inter-stage data flow strategy, the execution time of the data processing task according to the current inter-stage data flow strategy and the corresponding intra-stage data flow strategy is already less than a predetermined time threshold, it means that the execution time of the data processing task has been reduced to a level that meets the requirements, and there is no need to continue optimizing the inter-stage data flow strategy. Therefore, the execution time of the data processing task being less than the time threshold can be used as the optimization termination condition.

[0070] In this embodiment, execution cost convergence is used as the optimization termination condition. When executing data processing tasks based on the obtained inter-stage data flow strategy and the corresponding intra-stage data flow strategy, the cost of executing the data processing tasks can be minimized. Execution time of the data processing task is less than a time threshold is used as the optimization termination condition. When executing data processing tasks based on the obtained inter-stage data flow strategy and the corresponding intra-stage data flow strategy, the execution time of the data processing tasks can meet the requirements. Therefore, using execution cost convergence or execution time of the data processing task being less than a time threshold as the optimization termination condition can satisfy different requirements for data processing task execution, making it suitable for different application scenarios and improving the applicability of this data flow strategy generation scheme.

[0071] In one possible implementation, when updating the execution cost based on the current multi-stage data flow strategy, the execution cost within the multiple stages corresponding to the current multi-stage data flow strategy can be determined, and then the execution cost can be updated based on these multi-stage execution costs.

[0072] The intra-stage data flow strategy indicates the data processing scheme within the corresponding pipeline stage. Therefore, based on the intra-stage data flow strategy, the intra-stage execution cost of the corresponding pipeline stage can be determined. The intra-stage execution cost indicates the cost of data processing in the corresponding pipeline stage, such as time consumption cost and hardware resource consumption cost. The execution of data processing tasks is achieved through the comprehensive data processing within each pipeline stage. Therefore, the cost of executing data processing tasks is determined by the cost of data processing in each pipeline stage. Since the execution cost is based on the inter-stage data flow strategy and the corresponding intra-stage data flow strategy, the execution cost can be determined based on the intra-stage execution cost of each pipeline stage.

[0073] In one example, by pre-obtaining the intra-stage cost model and the inter-stage cost model, inputting the intra-stage data flow strategy into the intra-stage cost model allows for the calculation of the intra-stage execution cost for the corresponding pipeline stage. Inputting the intra-stage execution costs of each pipeline stage into the inter-stage cost model allows for the calculation of new execution costs. When calculating new execution costs using the inter-stage cost model, the current execution cost and the intra-stage execution costs of each pipeline stage can also be input into the inter-stage cost model to calculate the new execution cost.

[0074] In this embodiment, since the execution of the data processing task is achieved through data processing in multiple pipeline stages, the execution cost within the corresponding pipeline stage can be determined according to the data flow strategy within the stage. Then, a new execution cost is determined based on the execution cost within each pipeline stage, and the updated execution cost is obtained. This ensures that the updated execution cost can accurately reflect the cost of executing the data processing task based on the current inter-stage data flow strategy and the corresponding multiple intra-stage data flow strategies, thereby ensuring that the generated data flow strategy matches the requirements for executing the data processing task.

[0075] In one possible implementation, the execution cost includes hardware resources and task execution time. Accordingly, the inter-stage data flow strategy includes strategies for dividing the operators included in the computation graph into multiple pipeline stages, strategies for dividing hardware resources into multiple processing unit groups, and strategies for mapping multiple pipeline stages to multiple processing unit groups. Each pipeline stage includes at least one operator, and each processing unit group includes at least one processing unit.

[0076] When executing data processing tasks based on a pipelined architecture, the execution cost of the data processing task includes hardware resource consumption and execution time consumption. Under normal circumstances, the amount of hardware resources used in the execution of a data processing task is negatively correlated with the execution time of the data processing task. Therefore, the data flow strategy between stages can be optimized based on hardware resources and task execution time as execution costs. With the goal of minimizing hardware resource consumption or shortening task execution time, a data flow strategy including the target inter-stage data flow strategy and multiple target stage data flow strategies can be generated.

[0077] Hardware resources used to perform data processing tasks include the computing power, bandwidth, static random-access memory (SRAM), dynamic random-access memory (DRAM), and on-chip network (NoC). Multiple processing units (PEs) within these hardware resources are connected via an on-chip network. Figure 6 This is a schematic diagram of hardware resources according to one embodiment of this application. Figure 6 As shown, the hardware resources include multiple on-chip networks (NoCs) consisting of multiple processing units (PEs). Each on-chip network (NoC) consists of multiple processing units (PEs). Processing units (PEs) within the same on-chip network (NoC) communicate through the on-chip network (NoC). Processing units (PEs) within different on-chip networks (NoCs) communicate through an inter-on-chip network (NoC) communication network, such as Ethernet.

[0078] When hardware resources are divided into multiple processing unit groups, each processing unit group includes at least one processing unit (PE). When a processing unit group includes multiple processing units (PEs), the processing units (PEs) in the same processing unit group may be located in the same on-chip network (NoC) or in at least two different on-chip networks (NoCs), and this embodiment of the application does not limit this.

[0079] The processing unit group can be a wireless mesh network (PE Mesh). A PE Mesh consists of multiple PEs arranged in an array, and the divided pipeline stages can be mapped to a PE Mesh. The operators included in the corresponding pipeline stage are executed through the PE Mesh.

[0080] In this embodiment, the hardware resource partitioning strategy determines the number and computing power of the processing units included in the processing unit group, the pipeline stage partitioning strategy determines the operators included in the pipeline stage, and the strategy for mapping pipeline stages to processing unit groups determines the correspondence between pipeline stages and processing unit groups. When executing data processing tasks, the operators included in the pipeline stage can be deployed to the processing unit group corresponding to that pipeline stage, and the operators included in that pipeline stage can be executed by the processing units included in the processing unit group corresponding to that pipeline stage. The number and computing power of the processing units included in a processing unit group affect the execution speed of operators in the corresponding pipeline stage. Therefore, generating an inter-stage data flow strategy that includes a hardware resource partitioning strategy, a pipeline stage partitioning strategy, and a strategy for mapping pipeline stages to processing unit groups can determine the available computing resources for each pipeline stage based on the inter-stage data flow strategy. This allows for the determination of a better intra-stage data flow strategy for each pipeline stage, reducing the variable factors when determining the intra-stage data flow strategy, thereby shortening the time required to determine the intra-stage data flow strategy and improving the efficiency of generating the data flow strategy.

[0081] In one possible implementation, at least one of the hardware resources and task execution time included in the execution cost is updatable, meaning that when updating the execution cost according to the data flow strategy within a phase, at least one of the hardware resources and task execution time can be updated.

[0082] Depending on whether hardware resources and task execution time are updatable, execution costs can be categorized into three cases:

[0083] (i) Hardware resources are updatable, while task execution time is fixed.

[0084] (ii) Hardware resources are fixed items, while task execution time is an updatable item;

[0085] (iii) Both hardware resources and task execution time are updatable items.

[0086] Regarding the above situation (i), when updating the execution cost according to the intra-stage data flow strategy, the task execution time included in the execution cost remains unchanged, and only the hardware resources included in the execution cost are updated. This is suitable for scenarios where hardware resources are used sparingly, provided that the task execution time is less than the target duration. For example, a data processing task involves speech recognition using a speech recognition model. The model is required to output the speech recognition result within 0.5 seconds after receiving the input speech data. There is no limit to the consumption of hardware resources. When generating the data flow strategy for this data processing task, the hardware resource consumption is increased or decreased based on the hardware resource consumption and task execution time during speech recognition based on the current inter-stage data flow strategy and the corresponding intra-stage data flow strategy. Then, a new inter-stage data flow strategy is generated based on the updated hardware resources, thus updating the inter-stage data flow strategy and the corresponding intra-stage data flow strategy until the latency of the output speech recognition result is less than or equal to 0.5 seconds and the hardware resources are minimized.

[0087] When hardware resources are updatable and task execution time is fixed, available hardware resources are updated according to the intra-stage data flow strategy. If the task execution time is less than the target duration, the available hardware resources are reduced; if the task execution time is greater than the target duration, the available hardware resources are increased. The inter-stage data flow strategy is optimized by updating hardware resources until the task execution time is less than the target duration and the hardware resource consumption converges when executing data processing tasks based on the inter-stage data flow strategy and the corresponding intra-stage data flow strategy. The inter-stage data flow strategy and the corresponding multiple intra-stage data flow strategies at this time are determined as the target inter-stage data flow strategy and the corresponding multiple target intra-stage data flow strategies. A data flow strategy including the target inter-stage data flow strategy and each target intra-stage data flow strategy is obtained, and then the data flow strategy is used to execute the data processing task.

[0088] Regarding the above situation (ii), when updating the execution cost according to the intra-stage data flow strategy, the hardware resources included in the execution cost remain unchanged, and only the task execution time included in the execution cost is updated. This is suitable for scenarios where task execution time is short under the premise of limited hardware resources. For example, if the data processing task is the training task of a speech recognition model, and the allocable hardware resources are determined, to minimize the model training time, when generating the data flow strategy for this model training task, the task execution time is increased or decreased according to the hardware resource consumption and task execution time during model training based on the current inter-stage data flow strategy and the corresponding intra-stage data flow strategy. Then, a new inter-stage data flow strategy is generated based on the updated task execution time, realizing the updating of the inter-stage data flow strategy and the corresponding intra-stage data flow strategy until the training time of the model is minimized.

[0089] When hardware resources are fixed and task execution time is updatable, the task execution time is updated according to the intra-stage data flow strategy. The optimization goal is to minimize the task execution time. The inter-stage data flow strategy is optimized by updating the task execution time until the task execution time converges or is less than the target time when the data processing task is executed based on the inter-stage data flow strategy and the corresponding intra-stage data flow strategy. The inter-stage data flow strategy and the corresponding multiple intra-stage data flow strategies at this time are determined as the target inter-stage data flow strategy and the corresponding multiple intra-stage data flow strategies. A data strategy including the target inter-stage data flow strategy and each target intra-stage data flow strategy is obtained, and then the data flow strategy is used to execute the data processing task.

[0090] In scenario (iii) above, when updating the execution cost based on the intra-stage data flow strategy, the hardware resources and task execution time included in the execution cost can be updated. This is applicable to scenarios where both hardware resources and task execution time are unrestricted. For example, if the data processing task is the training task of a speech recognition model, and the available hardware resources and task execution time are unrestricted, the goal is to minimize the total cost of the model training task. The total cost is determined based on the hardware resource consumption cost and the task execution time cost. When generating the data flow strategy for this model training task, at least one of the hardware resources and task execution time is increased or decreased based on the hardware resource consumption and task execution time during model training, according to the current inter-stage data flow strategy and the corresponding intra-stage data flow strategy. Then, a new inter-stage data flow strategy is generated based on the updated hardware resources and task execution time, thus updating the inter-stage data flow strategy and the corresponding intra-stage data flow strategy until the total cost of executing the model training task is minimized.

[0091] When both hardware resources and task execution time are updatable, the hardware resources and task execution time are updated according to the intra-stage data flow strategy. The optimization objective is to minimize the total cost of the tasks to be processed. The inter-stage data flow strategy is optimized by updating the hardware resources and task execution time until the total cost of executing the data processing task is minimized or less than the cost threshold when the data processing task is executed based on the inter-stage data flow strategy and the corresponding intra-stage data flow strategy. The inter-stage data flow strategy and the corresponding multiple intra-stage data flow strategies at this time are determined as the target inter-stage data flow strategy and the corresponding multiple intra-stage data flow strategies. A data strategy including the target inter-stage data flow strategy and each target intra-stage data flow strategy is obtained, and then the data flow strategy is used to execute the data processing task.

[0092] In this application embodiment, depending on the different application functional scenarios, at least one of the hardware resources and task execution time included in the execution cost is an updatable item. In scenarios where the available hardware resources are fixed, the hardware resources are fixed items and the task execution time is an updatable item. In scenarios where the task execution time is required to be minimized or less than the target duration, both the hardware resources and the task execution time are updatable items. In scenarios where the total cost of task execution is required to be minimized or less than the cost threshold, both the hardware resources and the task execution time are updatable items. This makes the data flow strategy generation scheme provided in this application embodiment applicable to different application scenarios, improving the applicability of the data flow strategy generation scheme.

[0093] Figure 7 This is a flowchart of a phase-based data flow strategy generation method according to an embodiment of this application. Figure 7 As shown, the data flow strategy generation method in this stage includes the following steps:

[0094] Step 701: Generate multiple candidate data flow strategies for the operator based on the inter-stage data flow strategy.

[0095] The inter-stage data flow strategy includes strategies for dividing operators in the computation graph into multiple pipeline stages. Each pipeline stage includes at least one operator. Therefore, based on the inter-stage data flow strategy, the operators included in the pipeline stage can be determined. Then, based on the hardware resources allocated to the pipeline stage, the execution scheme of the operator in that pipeline stage can be determined, and candidate data flow strategies for the operator can be generated. The candidate data flow strategies include strategies for splitting the operator into multiple micro-operators, strategies for dividing the operator's tensor into multiple micro-tensors, and strategies for mapping micro-operators to processing units.

[0096] According to parallel schemes such as data parallelism and model parallelism, there are multiple execution schemes for operators. Therefore, for an operator, multiple candidate data flow strategies can be generated based on the inter-stage data flow strategy. Among the different candidate data flow strategies, at least one of the following is different: operator splitting strategy, tensor splitting strategy, and mapping strategy from micro-operator to processing unit.

[0097] Tensors are the input data of operators. After decomposing operators into multiple micro-operators, tensors need to be further divided into multiple micro-tensors. These micro-tensors are then used as inputs to the corresponding micro-operators, ensuring that the results of data processing through the decomposed micro-operators are the same as those processed through the original operators. When generating candidate data stream strategies, a tensor splitting strategy can be determined first. Once the tensor splitting strategy is determined, the corresponding operator splitting strategy can be determined, and then the mapping strategy from micro-operators to processing units can be determined.

[0098] The inter-stage data flow strategy includes a strategy for mapping pipeline stages to PE Mesh. PE Mesh consists of multiple PEs. After determining the operator splitting strategy, the mapping strategy from micro-operators to the corresponding PEs in the PE Mesh can be determined, and one or more PEs execute the mapped micro-operators.

[0099] Step 702: Determine the data processing cost of the candidate data stream strategy.

[0100] Operator splitting and tensor partitioning strategies determine the number of micro-operators that can be executed in parallel, thus affecting the data processing efficiency in the pipeline stage. The mapping strategy from micro-operators to processing units determines the time spent transmitting data between processing units, which in turn affects the data communication cost in the pipeline stage. Both the data processing efficiency and data communication cost in the pipeline stage affect the data processing cost. Therefore, the data processing cost of the candidate data flow strategy can be determined based on the operator splitting strategy, tensor partitioning strategy, and micro-operator-to-processing unit mapping strategy included in the candidate data flow strategy.

[0101] In one example, relevant information about candidate data stream strategies can be input into a cost calculation model, which can then be used to calculate the data processing cost of the candidate data stream strategies.

[0102] Step 703: Determine the intra-stage data flow strategy for the pipeline stage based on the data processing cost of multiple candidate data flow strategies for operators in the pipeline stage.

[0103] Based on the data processing costs corresponding to multiple candidate data flow strategies for an operator, a superior candidate data flow strategy can be selected from these strategies. Furthermore, the intra-stage data flow strategy for each operator in the pipeline stage can be determined based on the superior candidate data flow strategies. A pipeline stage includes one or more operators. If a pipeline stage includes one operator, its superior candidate data flow strategy can be used as the intra-stage data flow strategy. If a pipeline stage includes multiple operators, the superior candidate data flow strategies of each operator in the pipeline stage can be combined to obtain the intra-stage data flow strategy for that pipeline stage.

[0104] In this embodiment, multiple candidate data flow strategies can be generated for the operators included in the pipeline stage based on the inter-stage data flow strategy. The data processing cost of each candidate data flow strategy can be determined, and then, based on the data processing cost, a better combination of candidate data flow strategies is searched as the intra-stage data flow strategy for the pipeline stage. Generating data flow strategies on an operator-by-operator basis within the pipeline stage allows for rapid determination of candidate data flow strategies for each operator. Combining the better candidate data flow strategies for each operator to obtain the intra-stage data flow strategy for the pipeline stage improves the efficiency of generating data flow strategies for data processing task execution. Generating multiple candidate data flow strategies for operators and generating intra-stage data flow strategies by filtering and combining these strategies increases the probability of obtaining the target inter-stage and intra-stage data flow strategies.

[0105] In one possible implementation, when determining the intra-stage data flow strategy based on the data processing cost of the candidate data flow strategies, multiple candidate intra-stage data flow strategies for the pipeline stage can be generated based on the operators included in the pipeline stage and multiple candidate data flow strategies for the operators. Then, based on the data communication cost of the candidate data flow strategies included in the candidate intra-stage data flow strategies, the intra-stage data flow strategy for the pipeline stage is determined from the multiple candidate intra-stage data flow strategies. Wherein, different candidate intra-stage data flow strategies include at least partially different candidate data flow strategies.

[0106] After obtaining multiple candidate data flow strategies for each operator in the pipeline stage, these strategies can be combined to obtain multiple candidate intra-stage data flow strategies for that pipeline stage. Each candidate intra-stage data flow strategy must include at least one different candidate data flow strategy. For example, if the pipeline stage includes operator 0, operator 1, and operator 2, operator 0 has 2 candidate data flow strategies, operator 1 has 3, and operator 2 has 4. By combining the candidate data flow strategies of operator 0, operator 1, and operator 2, 2 × 3 × 4 candidate intra-stage data flow strategies can be obtained.

[0107] The execution cost of a data flow strategy within a candidate stage is determined by the execution cost of each operator included in that candidate stage's data flow strategy. In turn, the execution cost of an operator is determined by the candidate data flow strategy for that operator. Therefore, the intra-stage execution cost of each candidate stage's data flow strategy can be determined based on the data processing cost of each candidate stage's data flow strategy. Consequently, the intra-stage data flow strategy for the pipeline stage can be selected from the candidate stage's data flow strategies based on their intra-stage execution costs. For example, the candidate stage's data flow strategy with the lowest corresponding execution cost can be selected as the intra-stage data flow strategy for the corresponding pipeline stage.

[0108] In this embodiment, based on the operators included in the pipeline stage, candidate data flow strategies of each operator are combined to obtain multiple candidate intra-stage data flow strategies. Then, based on the data processing cost of the candidate data flow strategies included in the candidate intra-stage data flow strategies, the intra-stage execution cost of the candidate intra-stage data flow strategies is determined. Then, based on the intra-stage execution cost of each candidate intra-stage data flow strategy, the intra-stage data flow strategy of the pipeline stage is selected from the candidate intra-stage data flow strategies. This ensures that the determined intra-stage data flow strategy has a low execution cost, thereby enabling the target intra-stage data flow strategy to be obtained as quickly as possible and improving the efficiency of generating data flow strategies.

[0109] In one possible implementation, the candidate data flow strategy includes one of the following:

[0110] (i) A strategy of splitting an operator into multiple micro-operators according to the spatial dimension, wherein the multiple micro-operators split according to the spatial dimension are executed in parallel;

[0111] (ii) A strategy of splitting an operator into multiple micro-operators along the time dimension, wherein at least some of the micro-operators split along the time dimension are executed serially;

[0112] (iii) Strategy of splitting the operator into multiple micro-operators according to the spatial and temporal dimensions.

[0113] The candidate data stream includes operator splitting strategies, which are strategies for splitting an operator into multiple micro-operators. Depending on the dimension of the splitting operator, the operator splitting strategy can be a strategy of splitting the operator into multiple micro-operators according to the spatial dimension, or a strategy of splitting the operator into multiple micro-operators according to the temporal dimension, or a strategy of splitting the operator into multiple micro-operators according to both the spatial and temporal dimensions.

[0114] When an operator is decomposed into multiple micro-operators according to spatial dimensions, there are no data dependencies or control dependencies between the decomposed micro-operators, and each micro-operator can be executed in parallel. For example, OP0 can be decomposed into OP01, OP02, and OP03. OP01, OP02, and OP03 have no data dependencies or control dependencies on each other, and OP01, OP02, and OP03 can each perform data processing in parallel based on the input data. Figure 4 As shown, data parallelism is a method of operator parallelism that decomposes an operator into multiple micro-operators according to the spatial dimension. Figure 4 OP0, OP1, and OP2 are three micro-operators obtained by splitting an operator according to its spatial dimension.

[0115] When an operator is decomposed into multiple micro-operators along the time dimension, at least some of the decomposed micro-operators have data dependencies or control dependencies, and therefore at least some of the decomposed micro-operators are executed sequentially in a serial manner. For example, if OP0 is decomposed into OP01, OP02, OP03, and OP04, and OP02, OP03, and OP04 all have data dependencies or control dependencies on OP01, then OP02, OP03, and OP04 can only be executed after OP01 outputs its execution result. That is, OP01 and OP02, OP01 and OP03, and OP01 and OP04 are all executed serially. Figure 5 As shown, model parallelism is a method of operator parallelism that decomposes operators into multiple micro-operators along the time dimension. Figure 5 OP0, OP1, OP2, and OP3 are four micro-operators obtained by splitting an operator according to the time dimension.

[0116] Besides splitting operators individually according to spatial or temporal dimensions, operators can also be split according to both spatial and temporal dimensions. Some of the split micro-operators can be executed in parallel, while others can be executed sequentially. For example, OP0 can be split into OP01, OP02, OP03, OP04, and OP05. OP01, OP02, and OP03 can be executed in parallel, while OP03, OP04, and OP05 can be executed sequentially.

[0117] In this embodiment, the candidate data flow strategy for the operator may include a strategy of splitting the operator by spatial dimension, a strategy of splitting the operator by time dimension, or a strategy of splitting the operator by both spatial and time dimensions, splitting it into multiple micro-operators that can be executed in parallel, serially, or in both parallel and serial execution, thereby generating diverse candidate data flow strategies. Then, based on the candidate data flow strategies, a target stage data flow strategy that meets the requirements can be generated, improving the success rate of data flow strategy generation.

[0118] In one possible implementation, after obtaining the target inter-stage data flow strategy and the corresponding multiple intra-stage data flow strategies, inter-stage communication primitives between pipeline stages can be generated based on the target inter-stage data flow strategy, and intra-stage communication primitives within the corresponding pipeline stages can be generated based on the target intra-stage data flow strategy.

[0119] Since different pipeline stages may employ different tensor partitioning strategies, the partitioning strategy of tensors output by preceding pipeline stages may differ from the partitioning strategy of tensors input by subsequent pipeline stages. If the partitioning strategy of the input tensors does not match the pipeline stage, it will lead to errors in the computation results. Therefore, based on the target inter-stage data flow strategy, inter-stage communication primitives are generated between pipeline stages. These primitives are used to re-partition the tensors output by preceding pipeline stages before inputting them into subsequent pipeline stages. This ensures that the partitioning strategy of the input tensors in subsequent pipeline stages matches that of those stages, guaranteeing the correctness of the computation results.

[0120] Within the same pipeline stage, different operators can employ different tensor partitioning strategies, and different micro-operators can also use different tensor partitioning strategies. Therefore, the partitioning strategy of the tensor output by the preceding operator may differ from the partitioning strategy of the input tensor of the subsequent operator, and the partitioning strategy of the tensor output by the preceding micro-operator may differ from the partitioning strategy of the input tensor of the subsequent micro-operator. If the partitioning strategy of the input tensor does not match the operator / micro-operator, it will lead to errors in the computation results. To address this, based on the data flow strategy within the target stage, intra-stage communication primitives between operators and / or micro-operators are generated. These intra-stage communication primitives are used to re-partition the tensors output by the preceding operator / micro-operator before inputting them to the subsequent operator / micro-operator. This ensures that the partitioning strategy of the input tensor of the subsequent operator / micro-operator matches that of the subsequent operator / micro-operator, guaranteeing the accuracy of the computation results.

[0121] After generating inter-stage communication primitives and intra-stage communication primitives, these primitives can be attached to the target inter-stage data flow policy and the corresponding multiple intra-stage data flow policies to generate a data flow policy corresponding to the data processing task. Then, the data flow policy can be compiled into a machine-executable program by a compiler and deployed to the hardware to run, thereby realizing the execution of the data processing task.

[0122] In one possible implementation, during the data flow strategy generation process, the intra-stage data flow strategy can be optimized using an intra-stage optimizer, and the inter-stage data flow strategy can be optimized using an inter-stage optimizer. The following example illustrates the process of generating data flow strategies using intra-stage and inter-stage optimizers.

[0123] Figure 8 This is a schematic diagram of a data flow strategy generation process according to an embodiment of this application. The following is in conjunction with... Figure 8 The process of generating data flow strategies is explained.

[0124] In the first round of optimization, the inter-stage optimizer 801 generates inter-stage data flow strategies based on the computation graph and hardware resources, and sends these strategies to the intra-stage optimizer 802. The intra-stage optimizer 802, based on the inter-stage data flow strategies, calls the OP space splitting program 803 and the OP time splitting program 804 to split the operators within the pipeline stage. Based on the splitting results, it generates multiple intra-stage data flow strategies and calls the intra-stage cost model 805 to calculate the intra-stage execution cost of each inter-stage data flow strategy. The calculated intra-stage execution cost is then sent to the inter-stage optimizer 801. The OP space splitting program 803 is used to split operators into micro-operators according to the spatial dimension, and the OP time splitting program 804 is used to split operators into micro-operators according to the time dimension.

[0125] During the i-th round of optimization, where i is a positive integer greater than or equal to 2, the inter-stage optimizer 801 calculates the inter-stage execution cost based on the intra-stage execution cost received in the (i-1)-th round by calling the inter-stage cost model 806. Then, based on the computation graph, hardware resources, and the calculated inter-stage execution cost, it generates a new inter-stage data flow strategy and sends it to the intra-stage optimizer 802. The intra-stage optimizer 802, based on the newly received inter-stage data flow strategy, performs the same processing as in the first round, generates a new intra-stage data flow strategy, and sends the intra-stage execution cost of the new strategy to the inter-stage optimizer 801.

[0126] Following the optimization method of the i-th round of optimization process described above, after several rounds of optimization of the inter-stage data flow strategy and the intra-stage data flow strategy, if the inter-stage execution cost converges or the task execution time is less than the target duration, the inter-stage data flow strategy at this time is determined as the target inter-stage data flow strategy, and all the multiple intra-stage data flow strategies at this time are determined as the target intra-stage data flow strategies.

[0127] Based on the target intra-stage data flow strategy, intra-stage communication primitives are generated by the intra-stage communication primitive generator 807. Based on the target inter-stage data flow strategy, inter-stage communication primitives are generated by the inter-stage communication primitive generator 808. The intra-stage and inter-stage communication primitives are appended to the target intra-stage and inter-stage data flow strategies to obtain the data flow strategy. The data flow strategy is compiled into a machine-executable program by a compiler, and the machine-executable program is deployed to hardware for execution, thereby realizing the execution of data processing tasks.

[0128] It should be noted that the decisions of the inter-stage optimizer 801 can be optimized using appropriate algorithms, such as genetic algorithms, dynamic programming algorithms, and reinforcement learning algorithms.

[0129] Figure 9 This is a schematic diagram of the data flow strategy generation process according to another embodiment of this application. Figure 9 As shown, steps ① and ② are inter-stage optimization processes, while steps ③, ④, and ⑤ are intra-stage optimization processes.

[0130] Step ① is the step of dividing the pipeline into stages.

[0131] Step ② involves partitioning the PE Mesh and mapping the pipeline stages to the PE Mesh.

[0132] Step ③ is the process of splitting the operator (OP) into micro-operators (uOP) and mapping the micro-operators (uOP) to processing units (PE).

[0133] Step 4 involves dividing the tensor into microtensors and arranging the microtensors.

[0134] Step 5 is the step of communication arrangement between processing units (PE), that is, the step of inserting inter-stage communication primitives or intra-stage communication primitives between PEs.

[0135] In the example shown in step ③, the operator used to calculate matrix multiplication is split into multiple microoperators (uOPs). The microoperator C00_0 = A00 * B00 calculates the product of element A00 in Tensor A and element B00 in Tensor B; the microoperator C00_1 = A01 * B10 calculates the product of element A01 in Tensor A and element B10 in Tensor B; and the microoperator C00_0 = A00 * B00 sums C00_0 and C00_1. The elements of Tensor A and Tensor B can be called microtensors.

[0136] In the example shown in step ④, element A00 in Tensor A is programmed as the input of PE00 and PE01, element A10 in Tensor A is programmed as the input of PE10 and PE11, element B00 in Tensor B is programmed as the input of PE00 and PE10, and element B01 in Tensor B is programmed as the input of PE01 and PE11.

[0137] Data Stream Strategy Generation Device

[0138] Corresponding to the above method embodiments, Figure 10A schematic diagram of a data flow strategy generation apparatus according to an embodiment of this application is shown, such as... Figure 10 As shown, the data flow strategy generation device 1000 includes:

[0139] Acquisition unit 1001 is used to acquire the computation graph corresponding to the data processing task;

[0140] The first generation unit 1002 is used to generate an inter-stage data flow strategy based on the computation graph and execution cost, wherein the inter-stage data flow strategy includes a strategy of dividing the operators included in the computation graph into multiple pipeline stages, and each pipeline stage includes at least one operator.

[0141] The second generation unit 1003 is used to generate multiple intra-stage data flow strategies corresponding to multiple pipeline stages according to the inter-stage data flow strategy.

[0142] The update unit 1004 is used to update the execution cost according to the data flow strategy within multiple stages in order to optimize the data flow strategy between stages and obtain the target inter-stage data flow strategy and the corresponding multiple target intra-stage data flow strategies for performing data processing tasks.

[0143] In this embodiment, the data flow strategy includes an inter-stage data flow strategy and multiple intra-stage data flow strategies. The first generation unit 1002 can determine the inter-stage data flow strategy based on the computation graph and execution cost. The second generation unit 1003 can determine the intra-stage data flow strategy based on the inter-stage data flow strategy. The update unit 1004 can update the execution cost based on the intra-stage data flow strategy to optimize the inter-stage data flow strategy. The second generation unit 1003 can generate a new intra-stage data flow strategy based on the optimized inter-stage data flow strategy. Repeating the above optimization process can obtain a data flow strategy for data processing tasks. This data flow strategy includes a target inter-stage data flow strategy and multiple corresponding target intra-stage data flow strategies, thereby realizing the automated generation of data flow strategies. It can be applied to different data processing tasks and hardware resources and has strong applicability.

[0144] It should be noted that the data flow strategy generation device in this embodiment is used to implement the corresponding data flow strategy generation method in the foregoing method embodiment, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0145] electronic devices

[0146] Figure 11 This is a schematic block diagram of an electronic device provided in an embodiment of this application. Specific embodiments of this application do not limit the specific implementation of the electronic device. Figure 11As shown, the electronic device may include: a processor 1102, a communications interface 1104, a memory 1106, and a communication bus 1108. Wherein:

[0147] The processor 1102, communication interface 1104, and memory 1106 communicate with each other via communication bus 1108.

[0148] Communication interface 1104 is used to communicate with other electronic devices or servers.

[0149] The processor 1102 is used to execute program 1110, which can specifically execute the relevant steps in any of the aforementioned data flow strategy generation method embodiments.

[0150] Specifically, program 1110 may include program code that includes computer operation instructions.

[0151] The processor 1102 may be a CPU, an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application. The smart device may include one or more processors of the same type, such as one or more CPUs; or it may include processors of different types, such as one or more CPUs and one or more ASICs.

[0152] RISC-V is an open-source instruction set architecture based on the Reduced Instruction Set Computing (RISC) principle. It can be applied to various aspects of microcontrollers and FPGA chips, specifically in areas such as IoT security, industrial control, mobile phones, and personal computers. Because its design considers small size, speed, and low power consumption, it is particularly suitable for modern computing devices such as warehouse-scale cloud computers, high-end mobile phones, and tiny embedded systems. With the rise of AIoT (Artificial Intelligence of Things), the RISC-V instruction set architecture is receiving increasing attention and support and is expected to become the next generation of widely used CPU architecture.

[0153] The computer operation instructions in this application embodiment can be computer operation instructions based on the RISC-V instruction set architecture. Correspondingly, the processor 1102 can be designed based on the RISC-V instruction set. Specifically, the processor chip in the electronic device provided in this application embodiment can be a chip designed using the RISC-V instruction set. This chip can execute executable code based on the configured instructions, thereby implementing the data flow strategy generation method in the above embodiment.

[0154] Memory 1106 is used to store program 1110. Memory 1106 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0155] Specifically, program 1110 can be used to cause processor 1102 to execute the data flow strategy generation method in any of the foregoing embodiments.

[0156] The specific implementation of each step in program 1110 can be found in the corresponding steps and units described in any of the aforementioned data flow strategy generation method embodiments, and will not be repeated here. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the devices and modules described above can be referred to the corresponding process descriptions in the aforementioned method embodiments, and will not be repeated here.

[0157] The electronic device in this application embodiment provides a data flow strategy that includes an inter-stage data flow strategy and multiple intra-stage data flow strategies. The inter-stage data flow strategy can be determined based on the computation graph and execution cost. The intra-stage data flow strategy is determined based on the inter-stage data flow strategy. The execution cost can be updated based on the intra-stage data flow strategy to optimize the inter-stage data flow strategy. A new intra-stage data flow strategy can be generated based on the optimized inter-stage data flow strategy. Repeating the above optimization process can obtain a data flow strategy for data processing tasks. This data flow strategy includes a target inter-stage data flow strategy and multiple corresponding target intra-stage data flow strategies, thereby achieving automated generation of data flow strategies. It can be applied to different data processing tasks and hardware resources, and has strong applicability.

[0158] Computer storage media

[0159] This application also provides a computer-readable storage medium storing instructions for causing a machine to execute the data flow strategy generation method as described herein. Specifically, a system or apparatus equipped with a storage medium storing software program code that implements the functions of any of the embodiments described above, and enabling the computer (or CPU or MPU) of the system or apparatus to read and execute the program code stored in the storage medium.

[0160] In this case, the program code read from the storage medium can itself implement the function of any of the above embodiments, and therefore the program code and the storage medium storing the program code constitute part of this application.

[0161] Examples of storage media used to provide program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, program code can be downloaded from a server computer via a communication network.

[0162] Computer program products

[0163] This application also provides a computer program product, including computer instructions that instruct a computing device to perform any corresponding operation in the above-described plurality of method embodiments.

[0164] It should be noted that the user-related information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to sample data used for training the model, data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0165] It should be noted that, depending on the implementation needs, the various components / steps described in the embodiments of this application can be broken down into more components / steps, or two or more components / steps or parts of the operation of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of this application.

[0166] The methods described in the embodiments of this application can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code downloaded over a network that is originally stored in a remote recording medium or a non-transitory machine-readable medium and will be stored in a local recording medium. Thus, the methods described herein can be processed by software stored on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an ASIC or FPGA). It is understood that the computer, processor, microprocessor controller, or programmable hardware includes storage components (e.g., RAM, ROM, flash memory, etc.) capable of storing or receiving software or computer code that, when accessed and executed by the computer, processor, or hardware, implements the methods described herein. Furthermore, when a general-purpose computer accesses code used to implement the methods shown herein, the execution of the code transforms the general-purpose computer into a dedicated computer for executing the methods shown herein.

[0167] It should be noted that the user-related information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to sample data used for training the model, data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0168] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for specific applications, but such implementations should not be considered beyond the scope of the embodiments of this application.

[0169] The above embodiments are only used to illustrate the embodiments of this application, and are not intended to limit the embodiments of this application. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the embodiments of this application. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of this application, and the patent protection scope of the embodiments of this application should be defined by the claims.

Claims

1. A data flow strategy generation method, comprising: Obtain the computation graph corresponding to the data processing task; Based on the computation graph and execution cost, an inter-stage data flow strategy is generated, wherein the inter-stage data flow strategy includes a strategy of dividing the operators included in the computation graph into multiple pipeline stages, and each pipeline stage includes at least one operator. Based on the inter-stage data flow strategy, generate multiple intra-stage data flow strategies corresponding to the multiple pipeline stages; The execution cost is updated according to the multiple intra-stage data flow strategies to optimize the inter-stage data flow strategies, thereby obtaining a target inter-stage data flow strategy and a corresponding multiple target intra-stage data flow strategies for executing the data processing task. The step of generating multiple intra-stage data flow strategies corresponding to the multiple pipeline stages based on the inter-stage data flow strategy includes: Based on the inter-stage data flow strategy, multiple candidate data flow strategies for the operator are generated. The candidate data flow strategies include strategies for splitting the operator into multiple micro-operators, strategies for dividing the tensor of the operator into multiple micro-tensors, and strategies for mapping the micro-operators to processing units. Determine the data processing cost of the candidate data stream strategy; Based on the data processing cost of multiple candidate data flow strategies of the operators in the pipeline stage, the in-stage data flow strategy of the pipeline stage is determined, and the in-stage data flow strategy includes at least one candidate data flow strategy.

2. The method according to claim 1, wherein, The step of updating the execution cost according to the multiple intra-stage data flow strategies to optimize the inter-stage data flow strategies, and obtaining a target inter-stage data flow strategy and corresponding multiple target intra-stage data flow strategies for executing the data processing task, includes: The execution cost is updated based on the current data flow strategy across multiple phases. If the updated execution cost does not meet the optimization termination condition, then the step of generating the inter-stage data flow strategy based on the computation graph and execution cost is returned to optimize the current inter-stage data flow strategy. If the updated execution cost satisfies the optimization termination condition, then the current inter-stage data flow strategy and the corresponding multiple intra-stage data flow strategies are determined as the target inter-stage data flow strategy and the corresponding multiple target intra-stage data flow strategies.

3. The method according to claim 2, wherein, The optimization termination condition includes one of the following: (i) The execution cost converges; (ii) The execution time of the data processing task is less than the time threshold.

4. The method according to claim 2, wherein, The step of updating the execution cost based on the current data flow strategy within multiple stages includes: Determine the execution cost within multiple stages corresponding to the current data flow strategy within multiple stages; The execution cost is updated based on the execution cost within the multiple stages.

5. The method according to claim 1, wherein, The execution cost includes hardware resources and task execution time. The inter-stage data flow strategy includes a strategy of dividing the hardware resources into multiple processing unit groups and a strategy of mapping the multiple pipeline stages to the multiple processing unit groups. The processing unit group includes at least one processing unit.

6. The method according to claim 5, wherein, At least one of the hardware resources and the task execution time is an updatable item.

7. The method according to claim 1, wherein, The step of determining the in-stage data flow strategy for a pipeline stage based on the data processing cost of multiple candidate data flow strategies for operators in the pipeline stage includes: Based on the operators included in the pipeline stage and multiple candidate data flow strategies for the operators, multiple candidate intra-stage data flow strategies for the pipeline stage are generated, and the candidate intra-stage data flow strategies of different candidates include at least partially different candidate data flow strategies. Based on the data processing cost of the candidate data flow strategies included in the candidate stage data flow strategy, the stage data flow strategy of the pipeline stage is determined from the plurality of candidate stage data flow strategies.

8. The method according to claim 1, wherein, The candidate data stream strategy includes one of the following: (i) A strategy of splitting the operator into multiple micro-operators according to the spatial dimension, wherein the multiple micro-operators split according to the spatial dimension are executed in parallel; (ii) A strategy of splitting the operator into multiple micro-operators according to the time dimension, wherein at least some of the micro-operators split according to the time dimension are executed serially; (iii) A strategy of splitting the operator into multiple micro-operators according to the spatial and temporal dimensions.

9. The method according to any one of claims 1-8, further comprising: Based on the target inter-stage data flow strategy, generate inter-stage communication primitives between pipeline stages; Based on the data flow strategy within the target stage, generate the corresponding intra-stage communication primitives within the pipeline stage.

10. A data flow strategy generation apparatus, comprising: The acquisition unit is used to acquire the computation graph corresponding to the data processing task. The first generation unit is configured to generate an inter-stage data flow strategy based on the computation graph and execution cost, wherein the inter-stage data flow strategy includes a strategy of dividing the operators included in the computation graph into multiple pipeline stages, and the pipeline stage includes at least one operator. The second generation unit is used to generate multiple intra-stage data flow strategies corresponding to the multiple pipeline stages according to the inter-stage data flow strategy. The update unit is used to update the execution cost according to the multiple intra-stage data flow strategies to optimize the inter-stage data flow strategies and obtain the target inter-stage data flow strategy and the corresponding multiple target intra-stage data flow strategies for executing the data processing task. The second generation unit is configured to generate multiple candidate data flow strategies for the operator based on the inter-stage data flow strategy, wherein the candidate data flow strategies include strategies for splitting the operator into multiple micro-operators, strategies for splitting the tensor of the operator into multiple micro-tensors, and strategies for mapping the micro-operators to processing units; determine the data processing cost of the candidate data flow strategies; and determine the intra-stage data flow strategy of the pipeline stage based on the data processing costs of the multiple candidate data flow strategies for the operator in the pipeline stage, wherein the intra-stage data flow strategy includes at least one candidate data flow strategy.

11. An electronic device, comprising: The processor, memory, communication interface, and communication bus communicate with each other through the communication bus. The memory is used to store at least one executable instruction that causes the processor to perform the operation corresponding to any one of the methods in claims 1-9.

12. A computer storage medium having a computer program stored thereon, which, when executed by a processor, implements the method of any one of claims 1-9.

13. A computer program product comprising computer instructions that instruct a computing device to perform the method of any one of claims 1-9.

Images