Data processing method and device

Abstract

The invention discloses a data processing method, and the method comprises the steps: obtaining a plurality of different sub-structures, and enabling each sub-structure to indicate a data processing strategy of each calculation unit when a plurality of calculation units carry out the execution of a machine learning model through pipeline parallelism, and the relation between the data processing strategies of different calculation units, each substructure is a fragment of a complete execution process of the machine learning model for performing the pipeline parallelization; an optimal target substructure is determined from the multiple substructures through linear programming, a global processing strategy is constructed according to the target substructure, and the global processing strategy is a data processing strategy of the complete execution process; and executing the machine learning model through the plurality of computing units according to the global processing strategy. According to the method, the solving process is carried out on the fragments of the global processing strategy, so that the solving complexity can be reduced under the condition that the solving precision is ensured.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a communication system, a data processing method and an apparatus thereof. Background Technology

[0002] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0003] With the increasing computing power of computing devices, the improvement of communication network bandwidth, and the accumulation of data in big data networks, research is underway to deploy large-scale neural network models based on neural networks on distributed networks for computation and processing, in order to realize applications such as large-scale parallel data processing, distributed storage, elastic topology, high redundancy, and nonlinear operations in cloud computing.

[0004] Optimally distributing the training tasks of large models across different computing power cards while ensuring training stability is a significant challenge. To address the various challenges of distributed training, industry and academia have proposed numerous algorithms and systems. The most important aspect is determining the dimensions along which tasks are allocated, and how to achieve optimal allocation within fixed dimensions. The mainstream allocation dimensions in the industry include data parallelism, tensor parallelism, and pipelined parallelism. Pipeline parallelism addresses memory shortages by distributing parameters from different layers across different computing units, and compared to tensor parallelism, pipelined parallelism only requires point-to-point communication, resulting in lower communication overhead.

[0005] In pipeline parallelism, it is necessary to predetermine the data processing strategy for each computing unit. The data processing strategy can include the data processing order of each computing device for each batch of data, as well as the processing order of different computing units for the same batch of data. The data processing strategy needs to be solved through linear optimization. In the existing technology, in order to ensure the accuracy of the solution, the global data processing strategy is often solved, that is, the data processing strategy of the complete execution process of the machine learning model is solved. This leads to high computational complexity. As the scale of the problem increases, some commercial solvers on the market may not be able to complete the solution. Summary of the Invention

[0006] In a first aspect, this application provides a data processing method, the method comprising: acquiring multiple different substructures, each substructure indicating the data processing strategy of each computing unit and the relationship between the data processing strategies of different computing units when multiple computing units execute a machine learning model in pipeline parallelism, and each substructure being a segment of the complete pipeline parallel execution process of the machine learning model; determining the optimal target substructure from the multiple substructures through linear programming, and constructing a global processing strategy based on the target substructure, the global processing strategy being the data processing strategy of the complete execution process; and executing the machine learning model through the multiple computing units according to the global processing strategy.

[0007] In this embodiment, when determining the data processing strategy for executing the machine learning model through pipelined parallelism, the optimization is not performed globally. Considering that the global data processing strategy is often composed of multiple units (i.e., substructures) stacked (e.g., loops), this embodiment starts from the substructure, optimizes multiple different substructures to select a superior substructure (target substructure), and then executes the machine learning model based on the global processing strategy constructed from the target substructure. Since the solution process is performed on fragments of the global processing strategy, the complexity of the solution can be reduced while maintaining accuracy.

[0008] In one possible implementation, the fragment is specifically a loop unit of the complete pipelined parallel execution process of the machine learning model, the complete execution process being implemented by repeatedly executing the loop unit.

[0009] In one possible implementation, determining the optimal target substructure from the plurality of substructures through linear programming includes: determining a subset of substructures from the plurality of substructures as candidates for the optimal target substructure; and determining the optimal target substructure from the subset of substructures through linear programming.

[0010] In one possible implementation, to accelerate the solution process, a screening can be performed on multiple substructures to select the substructures with better performance as a candidate substructure set. Then, the optimal target substructure can be determined from the candidate substructure set by linear programming (e.g., mixed integer linear programming). In other words, the solution process can be further accelerated by reducing the search range of the optimal substructure.

[0011] In one possible implementation, determining a portion of the substructures from the plurality of substructures as candidates for the optimal target substructure includes: determining a portion of the substructures from the plurality of substructures as candidates for the optimal target substructure based on the performance of each of the substructures.

[0012] In one possible implementation, the plurality of computing devices includes a first computing device and a second computing device. The first computing device is deployed with a first network layer of the machine learning model, and the second computing device is deployed with a second network layer of the machine learning model. The second network layer is connected after the first network layer. The performance includes: in the relationship between the data processing strategies of different computing units indicated by the substructure, whether the data of the same batch is processed first through the first network layer on the first computing device and then through the second network layer on the second computing device.

[0013] Solve for an optimal substructure (Subscripts indicate the completion of a microbatch computation), where k∈[0,…N], and N is the maximum difference between the forward and backward propagation microbatch indices calculated using the memory constraints of each node. A good periodic substructure satisfies: k0≥k1…≥k 2D .

[0014] In one possible implementation, the performance metrics include the amount of time a computational unit is idle within the data processing strategy indicated by the substructure. A higher vacuolation rate in the substructure allows for more flexible subsequent arrangements, thus reducing the dependency between computational tasks within a single cycle substructure and facilitating the search for better solutions.

[0015] In one possible implementation, determining the optimal target substructure from the plurality of substructures through linear programming includes:

[0016] The optimal target substructure is determined from the plurality of substructures using mixed-integer linear programming.

[0017] In one possible implementation, the plurality of computing units can be a graphics processing unit (GPU), a network processing unit (NPU), or a tensor processing unit (TPU). Taking the plurality of computing units including a first computing unit and a second computing unit as an example, the first computing unit is deployed on a first node, and the second computing unit is deployed on a second node. The first node and the second node are different servers.

[0018] In one possible implementation, the first computing unit and the second computing unit are graphics processing units (GPUs), network processing units (NPUs), or tensor processing units (TPUs), and the first computing unit and the second computing unit are deployed on the same node.

[0019] Secondly, this application provides a data processing apparatus, the apparatus comprising:

[0020] The acquisition module is used to acquire multiple different substructures. Each substructure indicates the data processing strategy of each computing unit and the relationship between the data processing strategies of different computing units when multiple computing units execute the machine learning model in pipeline parallelism. Each substructure is a fragment of the complete pipeline parallel execution process of the machine learning model.

[0021] An optimization module is used to determine the optimal target substructure from the plurality of substructures through linear programming, and to construct a global processing strategy based on the target substructure, wherein the global processing strategy is the data processing strategy for the complete execution process;

[0022] The model execution module is used to execute the machine learning model through the multiple computing units according to the global processing strategy.

[0023] In one possible implementation, the fragment is specifically a loop unit of the complete pipelined parallel execution process of the machine learning model, the complete execution process being implemented by repeatedly executing the loop unit.

[0024] In one possible implementation, the optimization module is specifically used for:

[0025] From the plurality of substructures, select a subset of substructures as candidates for the optimal target substructure;

[0026] The optimal target substructure is determined from the substructures of the aforementioned part through linear programming.

[0027] In one possible implementation, the optimization module is specifically used for:

[0028] Based on the performance of each substructure, a subset of substructures are selected as candidates for the optimal target substructure from among the plurality of substructures.

[0029] In one possible implementation, the plurality of computing devices includes a first computing device and a second computing device, wherein the first computing device deploys a first network layer of the machine learning model, and the second computing device deploys a second network layer of the machine learning model, the second network layer being connected after the first network layer, and the performance characteristics include:

[0030] In the relationship between data processing strategies of different computing units indicated by the substructure, when processing data in the same batch, whether the data is first processed by the first network layer on the first computing device and then processed by the second network layer on the second computing device.

[0031] In one possible implementation, the performance characteristics include:

[0032] The substructure indicates the amount of time the computing unit is idle in the data processing strategy of each computing unit.

[0033] In one possible implementation, the optimization module is specifically used for:

[0034] The optimal target substructure is determined from the plurality of substructures using mixed-integer linear programming.

[0035] Thirdly, this application provides a data processing apparatus, which may include a memory, a processor, and a bus system, wherein the memory is used to store a program, and the processor is used to execute the program in the memory to perform the methods described in the first aspect above and any of its optional methods.

[0036] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the methods described in the first aspect and any of its optional methods.

[0037] Fifthly, embodiments of this application provide a computer program that, when run on a computer, causes the computer to perform the methods described in the first aspect and any of its alternatives.

[0038] Sixthly, this application provides a chip system including a processor for supporting the implementation of the functions involved in the foregoing aspects, such as transmitting or processing data or information involved in the foregoing methods. In one possible design, the chip system further includes a memory for storing program instructions and data necessary for the execution device or training device. This chip system may be composed of chips or may include chips and other discrete devices. Attached Figure Description

[0039] Figure 1 A structural diagram illustrating the main framework of artificial intelligence;

[0040] Figure 2 This is a schematic diagram of the application system framework of the present invention;

[0041] Figure 3 This is a schematic diagram of the application system framework in an embodiment of this application;

[0042] Figure 4 This is a schematic diagram of the application system framework in an embodiment of this application;

[0043] Figures 5A to 5I This is a schematic description of some operations in the embodiments of this application;

[0044] Figure 6 A flowchart illustrating a data processing method provided in an embodiment of this application;

[0045] Figure 7 This is a schematic diagram of the network structure in an embodiment of this application;

[0046] Figures 8A to 8D A flowchart illustrating a data processing method provided in an embodiment of this application;

[0047] Figure 9 This is a schematic diagram of the structure of the data processing apparatus according to an embodiment of this application;

[0048] Figure 10 This is a schematic diagram of a chip structure provided in an embodiment of this application. Detailed Implementation

[0049] The embodiments of the present invention will now be described with reference to the accompanying drawings. The terminology used in the embodiments section is for illustrative purposes only and is not intended to limit the scope of the invention.

[0050] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0051] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0052] The terms “substantially,” “about,” and similar terms used herein are used as approximations rather than as terms of degree, and are intended to take into account the inherent biases of measurements or calculations known to those skilled in the art. Furthermore, the use of “may” in describing embodiments of the invention refers to “one or more possible embodiments.” The terms “use,” “using,” and “used” used herein are to be considered synonymous with the terms “utilize,” “utilizing,” and “utilized,” respectively. Additionally, the term “exemplary” is intended to refer to an instance or illustration.

[0053] In some scenarios, large-scale neural network models based on neural networks can be deployed on distributed networks for computation and processing, enabling applications such as massive parallel processing, distributed storage, elastic topology, high redundancy, and nonlinear operations in cloud computing. Specifically, from the initial AlexNet network and VGG (visual geometry group network) to ResNet (residual neural network), GPT-3 (generative pre-trained transformer) and Pangu models, large-scale neural network models based on deep learning have become increasingly deeper, wider in scope, and have a larger number of parameters. Through these large-scale neural network models, massive computing and data resources can be utilized to achieve multi-task concurrent processing and large-scale computation.

[0054] However, when deploying large-scale neural network models on a system for computation, the rapid increase in model parameter size and data makes it difficult to perform complete training, inference, and other data processing on a single system. While deploying large-scale neural network models on a distributed system theoretically improves scalability and computational efficiency, the communication overhead—including data communication between nodes and communication between models—significantly increases when multiple models are processed in parallel across multiple nodes. This leads to reduced data transmission efficiency, resulting in decreased data processing efficiency, increased consumption of storage and computing resources, and higher application and maintenance costs. Furthermore, current large-scale neural network model architectures only consider deployment in densely connected networks (DenseNet), neglecting model sparsity and failing to support sparse architectures. In other words, it is impossible to activate only a portion of the large-scale neural network architecture to achieve computational processing for a given task, sample, or label.

[0055] It should be understood that the nodes in the embodiments of this application can be devices with the same granularity as the computing units, such as chips or AI cards. Alternatively, the computing units can be devices with finer granularity than the nodes. For example, a node can include multiple computing units. Specifically, a node can be a server, and the computing units can be chips or AI cards included in the server.

[0056] A mixture of experts (MoE) system, consisting of one or more expert modules, can be deployed across multiple nodes to build large-scale neural network models. When multiple expert modules of a MoE system are deployed at the same layer on multiple nodes, the input data of each node is independent and generally different. Further, the input data of each node is segmented at each node into data corresponding to each of the multiple expert modules, and each node sends the segmented data to the corresponding other nodes. Thus, each expert module on a node processes the data segmented from its own node and the data sent from other nodes. After all expert modules on each node have completed their data processing, the processed data are aggregated to generate the overall output of the neural network model.

[0057] Figure 1 A schematic diagram of an example system architecture 100 according to an embodiment of the present disclosure is shown. Figure 1 As shown, system architecture 100 may include at least a node cluster 110, a node server 120, client devices 130, and a network 140. The node cluster 110, node server 120, and client devices 130 in system architecture 100 can be connected via network 140. It should be understood that... Figure 1 The system architecture 100 may include more or fewer functional devices, and is not limited thereto.

[0058] In one example embodiment, client device 130 sends data to be processed to node cluster 110 and node server 120 via network 140 as input data for node cluster 110. Node server 120 controls node cluster 110 to process the data based on the data received from client device 130. Node cluster 110 processes the data received from client device 130 into a predetermined form through parallel processing and feeds it back to node server 120 or returns it to client device 130. In some embodiments, only a portion of the structure in system architecture 100 may participate in data processing. In one example embodiment, node server 120 may directly provide data to be processed to node cluster 110, while client device 130 does not participate in data processing. In one example embodiment, only a portion of the nodes in node cluster 110 may participate in data processing, while other nodes do not. It should be understood that the data processing operations between node cluster 110, node server 120, client device 130, and network 140 in system architecture 100 are not limited to these methods, and any operational approach can be implemented.

[0059] In some embodiments, node cluster 110 may include one or more nodes. For example, Figure 1 An example is illustrated where the node cluster 110 includes multiple nodes (node ​​110-1, node 110-2, node 110-3, ..., node 110-N). In the case of multiple nodes, the node cluster 110 can be distributed to form a neural network model, that is, the neural network model is deployed on multiple nodes of the node cluster 110 to process data in parallel.

[0060] It should be understood that the node cluster 110 may include multiple nodes, each node may include multiple computing units, or the node cluster 110 may include multiple computing units without any intermediate granularity device such as "node". For example, the node cluster 110 may be a server cluster, which may include multiple servers, each server may include multiple computing units. Or, for example, the node cluster 110 may be a server, and the server may include multiple computing units.

[0061] In some embodiments, the node server 120 can be used to manage and control the data processing of the node cluster 110, so as to determine the data processing algorithm or method of the node cluster 110 according to different data, or to use the node cluster 110 to perform training, derivation or other processing based on neural network models. In some embodiments, the node server 120 can be connected to all or a portion of the nodes of the node cluster 110, or it can be distributed across all or a portion of the nodes of the node cluster 110.

[0062] In some embodiments, client device 130 may be one or more suitable mobile or non-mobile computing devices for providing data input or receiving data feedback. Client device 130 may have data collection, processing, and output functions, such as input / output (I / O) devices like a recorder, camera, camcorder, mouse, keyboard, and monitor. Client device 130 may run various software applications, such as productivity or office support software, web browsing software, camera software, software supporting voice calls, video conferencing, and email. The computing device is connected to network 140 via wired and / or wireless communication links to communicate with node cluster 110 or node server 120.

[0063] In some embodiments, network 140 can be used to enable communication between various structures in system architecture 100, including node cluster 110, node server 120, and client device 130. For example, network 140 may include, for instance, a local area network (LAN), a wide area network (WAN), the Internet, a virtual LAN (VLAN), an enterprise LAN, a Layer 3 virtual private network (VPN), an intranet, or any combination thereof. It should be understood that the various structures in system architecture 100 can communicate directly with each other via network 140, or indirectly with each other via any structures connected to network 140, or they may not communicate with each other at all.

[0064] In this document, the term "input" refers to data sent to a device / unit / module / part, etc. The following example illustrates the use of image data as input data to the neural network model of node cluster 110. It should be understood that input data includes, but is not limited to, at least one of image data, text data, speech data, and classification data, as long as it is a data type suitable for processing by the neural network model. According to embodiments of this disclosure, it can be effectively applied to model training, inference, and processing of various types of data, including structured and unstructured data (e.g., image data, text data, speech data).

[0065] In this paper, the term "computing unit" refers to computing resources used for data processing, such as graphics processing units (GPUs), central processing units (CPUs), tensor processing units (TPUs), deep learning processing units (DPUs), neural network processing units (NPUs), and brain processing units (BPUs). It should be understood that the type and implementation of computing units are not limited to these; any computing unit suitable for processing neural network models is acceptable.

[0066] In this document, the term "parallel processing" means that data processing is performed in batches or stages at independent and asynchronous speeds, and the batches or stages of data processing may overlap in time. For example, the execution of one batch of data processing may not have finished on all nodes while the execution of another batch of data processing has already begun. It should be understood that parallel processing is not limited to what is listed herein, and any data processing of neural network models that can be understood by those skilled in the art is within the scope of parallel processing defined in the embodiments of this disclosure.

[0067] Figure 2 A schematic diagram of an example node cluster 110 according to an embodiment of the present disclosure is shown. For example, Figure 2 An example is shown as follows Figure 1 The node cluster 110 shown includes multiple nodes (node ​​210-1, node 210-2, node 210-3, ..., node 210-N), wherein the neural network model 200 is deployed across multiple nodes (node ​​210-1, node 210-2, node 210-3, ..., node 210-N), and the neural network model 200 includes P expert modules, where P is greater than or equal to 2. As an illustrative embodiment, the neural network model 200 may include 100 expert modules, which are distributed across multiple nodes (node ​​210-1, node 210-2, node 210-3, ..., node 210-N). Each node may deploy a portion of these 100 expert modules, and the expert modules included in each node collectively constitute the expert module whole of the neural network model 200. Figure 2As shown, in some embodiments, nodes 210-1, 210-2, 210-3, ..., 210-N are connected sequentially to perform data processing in a pipelined parallel manner. As an exemplary implementation of the pipelined parallel approach, for example, node 210-1 processes the input data of node cluster 110 and sends its output data (i.e., the intermediate data of node cluster 110 obtained after processing by node 210-1) to node 210-2. Then, node 210-2 processes the output data of node 210-1 and sends its output data (i.e., another intermediate data of node cluster 110 obtained after processing by node 210-2) to node 210-3. And so on, node 210-N processes the output data of the node connected to it and uses its output data (i.e., the data obtained after processing by node 210-N) as the output data of node cluster 110. It should be understood that the node connection method and data transmission method of multiple nodes in the node cluster 110 are not limited to this. The pipeline-based parallel method can be modified or extended on this basis as an additional or alternative method.

[0068] Figure 2 The node structure in node cluster 110 is also illustrated. For example, node 210-1 shows a node comprising one computing unit 20 and one computing unit 50, and node 210-3 shows a node comprising one computing unit 20 and two computing units 50, wherein computing unit 20 is a first computing unit based on expert parallelism, and computing unit 50 is a second computing unit based on tensor parallelism. It should be understood that the number of computing units 20 and 50 in each node can be arbitrary; each node may include only computing unit 20, only computing unit 50, or both computing units 20 and 50, and in some cases, may not include either computing unit 20 or computing unit 50.

[0069] Figure 3 A schematic diagram of a computing unit (e.g., a first computing unit or a second computing unit in an embodiment of this application) according to an embodiment of the present disclosure is shown. For example, Figure 3 An example is shown as follows Figure 1 and Figure 2A node in the node cluster 110 shown includes a computing unit 20 (i.e., a first computing unit). It should be understood that although a single node is used as an example here, the node structure described herein can be applied to any node in the node cluster 110, and in some embodiments, it can also be applied to other structures in the system architecture 100 (e.g., node server 120, client device 130, etc.).

[0070] like Figure 3 As shown, in some embodiments, each node includes one or more computing units, each of which can perform data processing physically independently. A node also includes one or more computing units, each of which can include one or more expert modules (e.g., fragments of expert modules) constituting a mixture of experts (MoE). Here, the mixture of experts system can be implemented by a single neural network or a combination of multiple neural networks, with each expert in the mixture of experts system implementing part or all of the neural network of the mixture of experts system. In some embodiments, the mixture of experts system can be deployed on one or more nodes or computing units. In some embodiments, each expert module can be used to perform a specific task and process a specific type of data in the node's input data. In some embodiments, a node also includes one or more storage resources (not shown) for storing intermediate processing data of the node, etc.

[0071] In some embodiments, a node may include multiple computing units, each computing unit including multiple expert modules E, wherein each expert module E may be a fragment of a complete expert module (i.e. a partial expert module), and expert modules on different computing units (e.g., different computing units on the same node, or computing units on different nodes) perform data processing in an expert parallel manner.

[0072] As an example implementation of an expert-parallel approach, for example, Figure 3 As shown, the node includes computing unit 11 and computing unit 12 (described as computing resources in the figure). Computing unit 11 and computing unit 12 perform data processing independently. For example, computing unit 11 includes expert modules E11 and E12, where E11 is a fragment of expert module A and E12 is a fragment of expert module B. Computing unit 12 includes expert modules E21 and E22, where E21 is another fragment of expert module A and E22 is another fragment of expert module B.

[0073] When data processing is required through expert module A, computing unit 11 can call E11 for data processing, and computing unit 12 can call E21 for data processing, and the two calculations are performed in parallel.

[0074] It should be understood that the number and relationship of computing units, computing units and expert modules E in this example are not limited thereto and do not constitute a limitation on the embodiments of this disclosure.

[0075] In some embodiments, each computing unit may include a gate unit. For example, such as Figure 3 As shown, the computing unit 11 includes a gating unit 31, and the processing resources include a gating unit 32. The gating unit is used to route the input data to the computing unit to determine which expert module (e.g., expert module A or expert module B) should execute the data.

[0076] For example, input data may also need to be segmented and processed in parallel by different computing units, such as... Figure 4 As shown, the system includes two nodes, each containing four computing units. The input features can be divided into eight parts, each processed by a computing unit on a node. In this case, after obtaining the input features, each computing unit can calculate which expert module needs to process the features of each token in the input features. If the expert module is deployed on the node where it is located, it can be retained (and subsequently synchronized through communication between computing units within the node, such as through the Allgather communication process). If the expert module is deployed on other nodes (i.e., not on its own node), data synchronization can be achieved through communication between computing units between nodes, such as through the AlltoAll communication process.

[0077] The following are some terms that will be used in the embodiments of this application:

[0078] 1) Reduce-scatter

[0079] This set communication primitive combines the reduce and scatter operations. For example... Figure 5A As shown, it performs a global reduce operation (such as summation, maximum value, etc.) on the data of an input array, and then distributes the results to various nodes.

[0080] 2) Allgather

[0081] Universal set communication is a many-to-many communication primitive, such as... Figure 5B As shown, it allows each node to send its data to all other nodes and collect data from all other nodes. After the AllGather operation, each node holds the complete dataset from all the nodes.

[0082] 3) Allreduce

[0083] AllReduce is a commonly used collection communication primitive in distributed computing, playing a crucial role, especially in distributed training within deep learning. This primitive is primarily used to synchronize data across multiple nodes or devices (such as an NPU). Figure 5C As shown, this typically involves aggregating data from multiple nodes and performing some form of reduction operation (such as summation, maximum value calculation, average value calculation, etc.), then broadcasting the result back to all nodes. In scenarios involving distributed training of large models, this primitive is mainly used for synchronizing activation values ​​during tensor-parallel forward and backward propagation, and for synchronizing gradient data in data-parallel systems. It's easy to see that the AllReduce primitive is equivalent to the superposition of the Reduce-scatter and Allgather communication primitives.

[0084] 4) AlltoAll

[0085] The AlltoAll operation allows each node to send its data to all other nodes and receive data from all other nodes. This is a many-to-many communication pattern that ensures each node receives data from all other nodes. This set communication primitive consists of two phases: data distribution and data aggregation. In the data distribution phase, on each node, the input data is split into `split_count` blocks according to the specified `split_dim` (split dimension). These blocks are then sent to other nodes in a certain order (e.g., block index); for example, the i-th block might be sent to the i-th node. In the data aggregation phase, each node receives data blocks from all other nodes. The received data blocks are then reassembled into a complete Tensor according to `concat_dim` (connection dimension). Tensors on all nodes must have the same size (shape) and format to ensure correct data aggregation. Figure 5D As shown, AlltoAll performs data transposition operations between distributed nodes.

[0086] 5) Tensor Parallelism

[0087] Tensor parallelism (TP) in distributed training of large models is a technique that divides the model parameters into multiple tensors and assigns each tensor to a different computing unit (such as an NPU) for computation.

[0088] For the Transformer architecture, tensor parallelism has specific applications in MLP layers and Attention layers. For example... Figure 5EAs shown on the right, taking the MLP layer as an example, the parameter matrix W1 is first "column-cut," and then the parameter matrix W2 is "row-cut" to perform parallel computation on two NPUs. During the forward propagation phase of training, according to the matrix operation rules, the function... The activation values ​​calculated on different devices need to be synchronized using an Allreduce operation to ensure the correctness of the forward computation. Similarly, during the backpropagation phase of training, the function f needs to be synchronized using an Allreduce operation on the gradients of the backpropagated activations to ensure the correctness of matrix operations.

[0089] like Figure 5E As shown on the left, the Attention layer is similar to the MLP layer, also first processing the parameter matrix W. Q W K and W V First, use "column cutting", then apply it to matrix W. L The "row-by-row" approach is used, therefore, Allreduce synchronization operations are introduced in both the forward and backward training phases. When using tensor parallelism, the overall computation and communication flow of the Transformer Block is as follows: Figure 5F As shown, there are 2 Allreduce steps forward and 2 Allreduce steps backward, for a total of 4 Allreduce steps.

[0090] 6) Megatron parallel pipeline

[0091] Megatron's Sequence Parallelism (SP) is an important technique in distributed training of large models, primarily to address the memory consumption issues in large Transformer models.

[0092] like Figure 5G As shown, taking the MLP module as an example, is the sequence splitting and tensor parallelism of model parameters consistent? Why? That is, the complete sequence is split into TP parts and placed on TP NPU cards respectively. During forward propagation, since each NPU card only has a portion of the sequence, before matrix operations, each NPU card in the TP domain needs to initiate Allgather communication to retrieve the sequences from other cards and concatenate them into a complete sequence for computation. After computation, the reduction operation required for tensor parallelism needs to be performed, and the data is distributed to each NPU card along the sequence dimension to restore the "original" sequence state. This involves calling Reduce-scatter communication. It's easy to understand that when tensor parallelism is further superimposed with Megatron's pipeline parallelism, the original Allreduce communication is split into Allgather and Reduce-scatter. Figure 5GAs shown, during the forward propagation phase of model training, the function g represents the Allgather operation required for computation by the Attention module and the MLP module, while the function... This refers to the Reduce-scatter operation; similarly, in the backpropagation phase, This indicates a Reduce-scatter operation, while the function g represents an Allgather operation. Normally, there are 2 Allgather operations and 2 Reduce-scatter operations in the forward direction, and 2 Allgather operations and 2 Reduce-scatter operations in the reverse direction, for a total of 4 Allgather operations and 4 Reduce-scatter operations.

[0093] 7) Parallel Expertise

[0094] Expert parallelism (EP) in distributed training of large models is a training strategy that allows the model to dynamically select a set of "experts" to perform computations based on the input data. This parallelization approach aims to optimize the use of computational resources by introducing sparsity, while maintaining or improving the model's performance.

[0095] The input data first passes through a gating network, which assigns weights or probabilities to different experts based on the input features. Then, based on these weights, the model selectively activates one or more experts to process the input data. Figure 5H As shown, since different experts may be placed on different devices (such as NPUs), and each device may contain data belonging to all experts, from the perspective of each device, it is necessary to send the data required by other experts on the current device to the corresponding device, and retrieve the data required for calculation by the experts on the current device stored on other devices. This operation can be achieved by initiating an All-to-All collection communication. Each expert independently processes its received data and produces output. These outputs are then returned to the original device and merged to produce the final output of the model. Therefore, it is necessary to...

[0096] To initiate the AlltoAll communication again. For example... Figure 5I As shown, during the forward propagation phase of model training, the function h represents the expression before and after expert parallelism. These represent dispatch and combine AlltoAll communication, respectively, and the reverse propagation phase works similarly.

[0097] 8) Parallel production lines

[0098] The problem of insufficient memory is solved by distributing parameters of different layers to different computing cards. At the same time, compared with tensor parallelism, pipeline parallelism only requires point-to-point communication, which has low communication overhead.

[0099] 9) Large-Scale Models: A type of large-scale machine learning model. Compared to simple linear models and traditional machine learning models, large-scale models construct complex nonlinear fitting functions by stacking a large number of multi-layered neurons to fit specific tasks. A commonly used basic module is a layer structure based on the attention mechanism. Generally, the first few layers of a large-scale model extract features from complex high-dimensional data, such as images, text, and communication data signals. The last layer outputs the target to be predicted; for example, in classification tasks, it outputs a score for each category. Finally, the loss is calculated based on the prediction results and the true labels, and the network parameters are updated end-to-end. Large-scale models have made remarkable progress in general artificial intelligence in recent years, achieving state-of-the-art results in many complex inference tasks and significantly outperforming other model architectures.

[0100] 10) Distributed Training: As the scale of machine learning models increases, a single computing card can no longer meet the training requirements of large models, which require a large amount of memory and computing power. Distributed training solves the problems of insufficient memory and excessively long training times by distributing the training tasks of large models across multiple computing cards to work together.

[0101] As the scale of machine learning models increases, traditional single-computing-card training is no longer sufficient. Training large models places extremely high demands on memory and computing power, posing many new technical challenges to distributed training. Optimally allocating the training tasks of large models across different computing cards while ensuring training stability is a significant challenge. To address these challenges, industry and academia have proposed numerous algorithms and systems. The most important approach is determining the dimensions along which tasks are allocated and how to achieve optimal allocation within fixed dimensions. The mainstream allocation dimensions in the industry include data parallelism, tensor parallelism, and pipelined parallelism. Pipeline parallelism addresses memory shortages by distributing parameters from different layers across different computing cards, and compared to tensor parallelism, pipelined parallelism only requires point-to-point communication, resulting in lower communication overhead.

[0102] In pipeline parallelism, it is necessary to predetermine the data processing strategy for each computing unit. The data processing strategy can include the data processing order of each computing device for each batch of data, as well as the processing order of different computing units for the same batch of data. The data processing strategy needs to be solved through linear optimization. In the existing technology, in order to ensure the accuracy of the solution, the global data processing strategy is often solved, that is, the data processing strategy of the complete execution process of the machine learning model is solved. This leads to high computational complexity. As the scale of the problem increases, some commercial solvers on the market may not be able to complete the solution.

[0103] To address the aforementioned problems, embodiments of this application provide a data processing method. The data processing method of this application embodiment will be described in detail below with reference to the accompanying drawings.

[0104] Reference Figure 6 , Figure 6 This is a flowchart illustrating a data processing method provided in an embodiment of this application, such as... Figure 6 As shown in the embodiment of this application, a data processing method may include steps 601 to 603, which are described in detail below.

[0105] 601. Obtain multiple different substructures, each substructure indicating the data processing strategy of each computing unit and the relationship between the data processing strategies of different computing units when multiple computing units execute the machine learning model in pipeline parallelism, and each substructure is a segment of the complete pipeline parallel execution process of the machine learning model.

[0106] In the process of pipeline parallelism, the model can be divided into pipelines first. Specifically, the entire model can be distributed (e.g., evenly distributed) to different computing units based on the forward and backward propagation times of each layer and the communication times between different layers obtained from full inference profiling.

[0107] When performing pipeline parallelism, it is necessary to predetermine the data processing strategy for each computing unit. The data processing strategy may include the data processing order of each computing device for each batch of data, and the processing order of different computing units for the same batch of data.

[0108] In one possible implementation, the fragment is specifically a loop unit of the complete pipelined parallel execution process of the machine learning model, the complete execution process being implemented by repeatedly executing the loop unit.

[0109] For example, you can refer to Figure 7 , Figure 7 This is a schematic diagram of the substructure, where different serial numbers represent different batches of data, and the time sequence of calculations is represented from left to right.

[0110] In this embodiment, when determining the data processing strategy for executing the machine learning model through pipelined parallelism, the optimization is not performed globally. Considering that the global data processing strategy is often composed of multiple units (i.e., substructures) stacked (e.g., loops), this embodiment starts from the substructure, optimizes multiple different substructures to select a superior substructure (target substructure), and then executes the machine learning model based on the global processing strategy constructed from the target substructure. Since the solution process is performed on fragments of the global processing strategy, the complexity of the solution can be reduced while maintaining accuracy.

[0111] The computing unit in this application embodiment may belong to a communication system. The communication system includes multiple computing units, such as a first computing unit and a second computing unit, and the first computing unit communicates with the second computing unit.

[0112] In one scenario, the communication system can be a collection of computing units, which can be chip-level, such as an NPU. In another scenario, the communication system can be a collection of nodes, each containing computing units. Nodes can be servers, and computing units can be chip-level, such as an NPU.

[0113] 602. By using linear programming, determine the optimal target substructure from the plurality of substructures, and construct a global processing strategy based on the target substructure, wherein the global processing strategy is the data processing strategy for the complete execution process;

[0114] In one possible implementation, the optimal target substructure can be determined from the plurality of substructures using mixed-integer linear programming.

[0115] In one possible implementation, to accelerate the solution process, a screening can be performed on multiple substructures to select the substructures with better performance as a candidate substructure set. Then, the optimal target substructure can be determined from the candidate substructure set by linear programming (e.g., mixed integer linear programming). In other words, the solution process can be further accelerated by reducing the search range of the optimal substructure.

[0116] In one possible implementation, a subset of substructures is determined from the plurality of substructures as candidates for the optimal target substructure, and the optimal target substructure is determined from the subset of substructures by linear programming.

[0117] In one possible implementation, a subset of the substructures can be determined as candidates for the optimal target substructure based on the performance of each substructure.

[0118] In one possible implementation, the plurality of computing devices includes a first computing device and a second computing device. The first computing device is deployed with a first network layer of the machine learning model, and the second computing device is deployed with a second network layer of the machine learning model. The second network layer is connected after the first network layer. The performance includes: in the relationship between the data processing strategies of different computing units indicated by the substructure, whether the data of the same batch is processed first through the first network layer on the first computing device and then through the second network layer on the second computing device.

[0119] In one possible implementation, the performance characteristics include: the amount of time a computing unit is idle in the data processing strategy of each computing unit indicated by the substructure.

[0120] For example, based on monotonicity and the theoretical lower bound of different substructures, all possible alternatives that could become the optimal periodic substructure can be traversed.

[0121] The following provides a further illustrative example of the details of optimal substructure pruning, solving for an optimal substructure. (Subscripts indicate the completion of a microbatch computation), where k∈[0,…N], and N is the maximum difference between the forward and backward propagation microbatch indices calculated using the memory constraints of each node. A good periodic substructure satisfies two important properties: 1) k0≥k1…≥k 2D (2) k0 and k 2D The difference should be as large as possible to weaken the dependency between computational tasks within a periodic substructure, thus facilitating the search for better solutions. Two important properties can be used to significantly reduce the search space and filter out potential candidate periodic substructures. Figure 8A The two types of bad periodic substructures are shown.

[0122] For example, given a candidate periodic substructure, it can be solved by... Figure 8B This small-scale mixed-integer linear programming problem is complete.

[0123] Finding the optimal substructure allows for expansion to a global scheduling scheme. Since steady-state operations account for a significant portion of training time, this method of reducing the problem size ensures the approximate optimality of the solution. Our scheme yields the optimal solution when MBS approaches infinity.

[0124] In existing technologies, PP scheduling optimization is often based on simple heuristic rules, which cannot guarantee the optimality of the solution. This solution significantly reduces the solution complexity of the original scheduling problem by deeply exploring the mathematical properties of the optimal substructure of the large model during the steady state, and guarantees the optimality of the solution when MBS tends to infinity. Since MBS is often very large in real-world scenarios, often reaching thousands, the theoretical properties of the embodiments in this application are ideal.

[0125] In one possible implementation, a global processing strategy can be obtained by iterating through the target substructure. This global processing strategy is the data processing strategy for the complete execution process.

[0126] It should be understood that, optionally, the scheduling scheme can be solved by splicing the optimal sub-cycles during steady-state training, while the startup and cooling states can be solved using a 1F1B-like heuristic algorithm.

[0127] 603. According to the global processing strategy, the machine learning model is executed through the multiple computing units.

[0128] Reference Figure 8C , Figure 8C This is an example diagram of an application scenario of an embodiment of this application, which includes hardware modules: N computing power cards; the machine learning model used can be a large language model, such as Llama and Pangu based on attention architecture.

[0129] Reference Figure 8D , Figure 8D This is a schematic diagram of an application process according to an embodiment of this application.

[0130] Reference Figure 9 , Figure 9 This application provides a schematic diagram of the structure of a data processing apparatus, which includes:

[0131] The acquisition module 901 is used to acquire multiple different substructures. Each substructure indicates the data processing strategy of each computing unit and the relationship between the data processing strategies of different computing units when multiple computing units execute the machine learning model in pipeline parallelism. Each substructure is a segment of the complete pipeline parallel execution process of the machine learning model.

[0132] The optimization module 902 is used to determine the optimal target substructure from the plurality of substructures through linear programming, and to construct a global processing strategy based on the target substructure, wherein the global processing strategy is the data processing strategy for the complete execution process;

[0133] The model execution module 903 is used to execute the machine learning model through the multiple computing units according to the global processing strategy.

[0134] In one possible implementation, the fragment is specifically a loop unit of the complete pipelined parallel execution process of the machine learning model, the complete execution process being implemented by repeatedly executing the loop unit.

[0135] In one possible implementation, the optimization module is specifically used for:

[0136] From the plurality of substructures, select a subset of substructures as candidates for the optimal target substructure;

[0137] The optimal target substructure is determined from the substructures of the aforementioned part through linear programming.

[0138] In one possible implementation, the optimization module is specifically used for:

[0139] Based on the performance of each substructure, a subset of substructures are selected as candidates for the optimal target substructure from among the plurality of substructures.

[0140] In one possible implementation, the plurality of computing devices includes a first computing device and a second computing device, wherein the first computing device deploys a first network layer of the machine learning model, and the second computing device deploys a second network layer of the machine learning model, the second network layer being connected after the first network layer, and the performance characteristics include:

[0141] In the relationship between data processing strategies of different computing units indicated by the substructure, when processing data in the same batch, whether the data is first processed by the first network layer on the first computing device and then processed by the second network layer on the second computing device.

[0142] In one possible implementation, the performance characteristics include:

[0143] The substructure indicates the amount of time the computing unit is idle in the data processing strategy of each computing unit.

[0144] In one possible implementation, the optimization module is specifically used for:

[0145] The optimal target substructure is determined from the plurality of substructures using mixed-integer linear programming.

[0146] This application also provides a computer program product that, when run on a computer, causes the computer to perform steps as performed by the aforementioned execution device, or causes the computer to perform steps as performed by the aforementioned training device.

[0147] This application also provides a computer-readable storage medium storing a program for signal processing, which, when run on a computer, causes the computer to perform steps as performed by the aforementioned execution device, or causes the computer to perform steps as performed by the aforementioned training device.

[0148] The execution device, training device, or terminal device provided in this application embodiment can specifically be a chip. The chip includes a processing unit and a communication unit. The processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in the storage unit to cause the chip within the execution device to execute the data processing method described in the above embodiments, or to cause the chip within the training device to execute the data processing method described in the above embodiments. Optionally, the storage unit can be a storage unit within the chip, such as a register or cache. Alternatively, the storage unit can be a storage unit located outside the chip within the wireless access device, such as a read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, such as random access memory (RAM).

[0149] For details, please refer to Figure 10 , Figure 10 This is a schematic diagram of a chip provided in an embodiment of this application. The chip can be represented as a neural network processor (NPU) 1000. The NPU 1000 is mounted as a coprocessor on the host CPU, and tasks are assigned by the host CPU. The core part of the NPU is the arithmetic circuit 1003, which is controlled by the controller 1004 to extract matrix data from the memory and perform multiplication operations.

[0150] In some implementations, the arithmetic circuit 1003 internally includes multiple processing engines (PEs). In some implementations, the arithmetic circuit 1003 is a two-dimensional pulsating array. The arithmetic circuit 1003 can also be a one-dimensional pulsating array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 1003 is a general-purpose matrix processor.

[0151] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory 1002 and caches it in each PE of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory 1001 and performs matrix operations with matrix B. The partial result or the final result of the obtained matrix is ​​stored in the accumulator 1008.

[0152] Unified memory 1006 is used to store input and output data. Weight data is directly transferred to weight memory 1002 via Direct Memory Access Controller (DMAC) 1005. Input data is also transferred to unified memory 1006 via DMAC.

[0153] BIU stands for Bus Interface Unit, which is used for interaction between the AXI bus and the DMAC and the Instruction Fetch Buffer (IFB) 1009.

[0154] The Bus Interface Unit (BIU) 1010 is used by the instruction fetch memory 1009 to fetch instructions from external memory, and also by the memory access controller 1005 to fetch the original data of the input matrix A or the weight matrix B from external memory.

[0155] The DMAC is mainly used to move input data from external memory DDR to unified memory 1006, or to weight data to weight memory 1002, or to input data to input memory 1001.

[0156] The vector computation unit 1007 includes multiple processing units that further process the output of the computation circuit 1003 when needed, such as vector multiplication, vector addition, exponential operations, logarithmic operations, size comparisons, etc. It is mainly used for computation in non-convolutional / fully connected layers of neural networks, such as batch normalization, pixel-level summation, and upsampling of feature planes.

[0157] In some implementations, the vector computation unit 1007 can store the processed output vector in the unified memory 1006. For example, the vector computation unit 1007 can apply a linear function, or a nonlinear function, to the output of the computation circuit 1003, such as performing linear interpolation on the feature planes extracted by the convolutional layer, or, for example, accumulating a vector of values ​​to generate activation values. In some implementations, the vector computation unit 1007 generates normalized values, pixel-level summed values, or both. In some implementations, the processed output vector can be used as an activation input to the computation circuit 1003, for example, for use in subsequent layers of the neural network.

[0158] The instruction fetch buffer 1009 connected to the controller 1004 is used to store the instructions used by the controller 1004;

[0159] Unified memory 1006, input memory 1001, weighted memory 1002, and instruction fetch memory 1009 are all on-chip memories. External memory is proprietary to this NPU hardware architecture.

[0160] The processor mentioned above can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits used to control the execution of the above program.

[0161] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the accompanying drawings of the device embodiments provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0162] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods of the various embodiments of this application.

[0163] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0164] A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

Claims

1. A data processing method, characterized in that, The method includes: Multiple different substructures are obtained. Each substructure indicates the data processing strategy of each computing unit and the relationship between the data processing strategies of different computing units when multiple computing units execute the machine learning model in pipeline parallelism. Each substructure is a fragment of the complete pipeline parallel execution process of the machine learning model. By using linear programming, the optimal target substructure is determined from the plurality of substructures, and a global processing strategy is constructed based on the target substructure. The global processing strategy is the data processing strategy for the complete execution process. According to the global processing strategy, the machine learning model is executed through the multiple computing units.

2. The method according to claim 1, characterized in that, The segment is specifically a loop unit of the complete execution process of the machine learning model performing the pipelined parallel execution, and the complete execution process is achieved by repeatedly executing the loop unit.

3. The method according to claim 1 or 2, characterized in that, The step of determining the optimal target substructure from the plurality of substructures through linear programming includes: From the plurality of substructures, select a subset of substructures as candidates for the optimal target substructure; The optimal target substructure is determined from the substructures of the aforementioned part through linear programming.

4. The method according to any one of claims 1 to 3, characterized in that, The step of determining a subset of the substructures from the plurality of substructures as candidates for the optimal target substructure includes: Based on the performance of each substructure, a subset of substructures are selected as candidates for the optimal target substructure from among the plurality of substructures.

5. The method according to claim 4, characterized in that, The plurality of computing devices includes a first computing device and a second computing device. The first computing device deploys a first network layer of the machine learning model, and the second computing device deploys a second network layer of the machine learning model. The second network layer is connected after the first network layer. The performance characteristics include: In the relationship between data processing strategies of different computing units indicated by the substructure, when processing data in the same batch, whether the data is first processed by the first network layer on the first computing device and then processed by the second network layer on the second computing device.

6. The method according to claim 4, characterized in that, The performance characteristics include: The substructure indicates the amount of time the computing unit is idle in the data processing strategy of each computing unit.

7. The method according to any one of claims 1 to 6, characterized in that, The step of determining the optimal target substructure from the plurality of substructures through linear programming includes: The optimal target substructure is determined from the plurality of substructures using mixed-integer linear programming.

8. A data processing apparatus, characterized in that, The device includes: The acquisition module is used to acquire multiple different substructures. Each substructure indicates the data processing strategy of each computing unit and the relationship between the data processing strategies of different computing units when multiple computing units execute the machine learning model in pipeline parallelism. Each substructure is a fragment of the complete pipeline parallel execution process of the machine learning model. An optimization module is used to determine the optimal target substructure from the plurality of substructures through linear programming, and to construct a global processing strategy based on the target substructure, wherein the global processing strategy is the data processing strategy for the complete execution process; The model execution module is used to execute the machine learning model through the multiple computing units according to the global processing strategy.

9. The apparatus according to claim 8, characterized in that, The segment is specifically a loop unit of the complete execution process of the machine learning model performing the pipelined parallel execution, and the complete execution process is achieved by repeatedly executing the loop unit.

10. The apparatus according to claim 8 or 9, characterized in that, The optimization module is specifically used for: From the plurality of substructures, select a subset of substructures as candidates for the optimal target substructure; The optimal target substructure is determined from the substructures of the aforementioned part through linear programming.

11. The apparatus according to any one of claims 8 to 10, characterized in that, The optimization module is specifically used for: Based on the performance of each substructure, a subset of substructures are selected as candidates for the optimal target substructure from among the plurality of substructures.

12. The apparatus according to claim 11, characterized in that, The plurality of computing devices includes a first computing device and a second computing device. The first computing device deploys a first network layer of the machine learning model, and the second computing device deploys a second network layer of the machine learning model. The second network layer is connected after the first network layer. The performance characteristics include: In the relationship between data processing strategies of different computing units indicated by the substructure, when processing data in the same batch, whether the data is first processed by the first network layer on the first computing device and then processed by the second network layer on the second computing device.

13. The apparatus according to claim 11, characterized in that, The performance characteristics include: The substructure indicates the amount of time the computing unit is idle in the data processing strategy of each computing unit.

14. The apparatus according to any one of claims 8 to 13, characterized in that, The optimization module is specifically used for: The optimal target substructure is determined from the plurality of substructures using mixed-integer linear programming.

15. A computer storage medium, characterized in that, The computer storage medium stores one or more instructions that, when executed by one or more computers, cause the one or more computers to perform the operation of the method according to any one of claims 1 to 7.

16. A computer program product, characterized in that, Includes computer-readable instructions that, when executed on a computer device, cause the computer device to perform the method as described in any one of claims 1 to 7.

17. A system comprising at least one processor and at least one memory; the processor and the memory are connected via a communication bus and communicate with each other. The at least one memory is used to store code; The at least one processor is used to execute the code to perform the method as described in any one of claims 1 to 7.

18. A chip, characterized in that, It includes at least one processing unit and an interface circuit, the interface circuit being used to provide program instructions or data to the at least one processing unit, the at least one processing unit being used to execute the program instructions to implement the method of any one of claims 1 to 7.

Images