Parallel processing method, parallel processing device, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses a parallel processing method, a parallel processing device, electronic equipment and a storage medium, which are used for improving the model operation efficiency and improving the hardware utilization rate of a processor. The parallel processing method comprises the following steps: segmenting input data in a target model to obtain a first segmented data set and a second segmented data set; respectively processing the first slice data set and the second slice data set through a parallel network module to obtain a first branch sequence and a second branch sequence; determining an inter-branch parallel operator set from the first branch sequence and the second branch sequence through dynamic programming, wherein the inter-branch parallel operator set comprises a first operator from the first branch sequence and a second operator from the second branch sequence; and executing the first operator and the second operator in parallel to obtain an operator execution result.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to a parallel processing method, parallel processing device, electronic device, and storage medium. Background Technology

[0002] For large-scale models in deep learning, the number of parameters can reach trillions. Current artificial intelligence algorithms rely on neural networks, such as GPT-3 and GPT-4, which have thousands or even trillions of parameters. However, large-scale models have become extremely expensive due to their enormous computational cost. To address this issue, deep learning technology has begun to focus on Mixture of Experts (MoE) architectures, providing a way to increase the number of model parameters without increasing computational costs.

[0003] Large-scale models in the MoE architecture typically have a large number of parameters, requiring additional expert parallelism for model partitioning. However, the expert parallelism strategy introduces time-consuming all-to-all (A2A) communication operators; as the cluster size increases, the proportion of A2A communication also increases, becoming a bottleneck for model computation.

[0004] Currently, more efficient communication protocols and networks are being used to alleviate the problem of high computational load in models to some extent, but the problems of low computational efficiency and low processor hardware utilization still exist. Summary of the Invention

[0005] This application provides a parallel processing method, parallel processing device, electronic device, and storage medium to improve model computation efficiency and increase processor hardware utilization.

[0006] To address the aforementioned technical problems, this application provides the following technical solutions:

[0007] In a first aspect, embodiments of this application provide a parallel processing method, comprising: segmenting input data in a target model to obtain a first segmented data set and a second sliced ​​data set; processing the first segmented data set and the second sliced ​​data set respectively through a parallel network module to obtain a first branch sequence and a second branch sequence, wherein the first branch sequence and the branch sequence respectively include multiple operators; determining a set of inter-branch parallel operators from the first branch sequence and the second branch sequence through dynamic programming, wherein the set of inter-branch parallel operators includes: a first operator from the first branch sequence and a second operator from the second branch sequence; and executing the first operator and the second operator in parallel to obtain operator execution results.

[0008] In the above scheme, the input data in the target model is segmented to obtain a first segmented data set and a second slice data set. The first segmented data set and the second slice data set are processed by a parallel network module to form two branch sequences that can be executed in parallel. Since most operators in the branch sequences do not have dependencies, the first and second operators in the two branch sequences that do not have dependencies are scheduled in parallel to improve the computational utilization of the processor and reduce the training and inference time of the target model.

[0009] In one possible implementation of the first aspect, the method further includes: removing commonly accessed operators from the first branch sequence and the second branch sequence to obtain a first branch sequence and a second branch sequence after removing the operators;

[0010] The step of determining the set of inter-branch parallel operators from the first branch sequence and the second branch sequence through dynamic programming includes:

[0011] The set of parallel operators between branches is determined by dynamic programming from the first branch sequence after the removal operator and the second branch sequence after the removal operator.

[0012] In the above scheme, by removing operators that are accessed in common from the two branch sequences, operators with dependencies are removed from the two branch sequences. This ensures that there are no dependencies when operators are executed in parallel in the two branch sequences, thus avoiding errors during model runtime.

[0013] In one possible implementation of the first aspect, removing the commonly accessed operator from the first branch sequence and the second branch sequence includes:

[0014] Starting with the merge operator inserted into the target model, branch label propagation is performed on the first branch sequence and the second branch sequence, and branch label propagation stops at the splitting operator inserted into the target model.

[0015] In the above scheme, starting from the merging operator, the label propagation is performed in reverse according to the first branch sequence and the second branch sequence respectively. For operators that are accessed in both the first branch sequence and the second branch sequence, the label propagation will not be performed. That is, the operators that are accessed in both branches can be removed from the first branch sequence and the second branch sequence. The branch label propagation stops when the label propagation reaches the splitting operator.

[0016] In one possible implementation of the first aspect, the method further includes: merging consecutive operators of the same type within each of the first branch sequence and the second branch sequence to obtain a first branch sequence after merging operators and a second branch sequence after merging operators;

[0017] The step of determining the set of inter-branch parallel operators from the first branch sequence and the second branch sequence through dynamic programming includes:

[0018] The set of parallel operators between branches is determined by dynamic programming from the first branch sequence after the merging operator and the second branch sequence after the merging operator.

[0019] In the above scheme, the operators within the sequence are merged in the first branch sequence and the operators within the sequence are merged in the second branch sequence. Then, dynamic programming can be performed on the first branch sequence after merging the operators and the second branch sequence after merging the operators. Since dynamic programming is performed on the two branch sequences after merging the operators, the efficiency of generating the set of parallel operators between branches can be improved.

[0020] In one possible implementation of the first aspect, merging consecutive operators of the same type within each of the first and second branch sequences includes:

[0021] Obtain the runtime of the operators within each sequence of the first branch sequence and the second branch sequence;

[0022] Obtain the sum of runtime of consecutive operators of the same type within each of the first branch sequence and the second branch sequence;

[0023] If the total runtime is less than a first runtime threshold, consecutive operators of the same type within the first branch sequence and the second branch sequence are merged.

[0024] In the above scheme, if the sum of the runtime of consecutive operators of the same type in the first branch sequence and the second branch sequence is less than the first time threshold, the consecutive operators of the same type in the first branch sequence and the second branch sequence are merged to avoid operators with excessive runtime after merging, thereby improving the execution efficiency of operators in the target model.

[0025] In one possible implementation of the first aspect, determining the set of inter-branch parallel operators from the first branch sequence and the second branch sequence through dynamic programming includes:

[0026] Obtain the first hidden duration for communication and computation between the first (i-1) operators in the first branch sequence and the first j operators in the second branch sequence, where i and j are positive integers;

[0027] Obtain the second hidden duration for communication and computation between the first i operators in the first branch sequence and the first (j-1) operators in the second branch sequence;

[0028] Obtain the third hidden duration for communication and computation between the first (i-1) operators in the first branch sequence and the first (j-1) operators in the second branch sequence, and the fourth hidden duration for communication and computation between the i-th operator in the first branch sequence and the j-th operator in the second branch sequence;

[0029] A fifth hidden duration is obtained based on the first hidden duration, the second hidden duration, the third hidden duration, and the fourth hidden duration. The fifth hidden duration is the hidden duration for communication and calculation between the first i operators in the first branch sequence and the first j operators in the second branch sequence.

[0030] The maximum duration of the fifth hidden duration is taken as the optimal path of the dynamic programming, and the i-th operator in the first branch sequence is determined as the first operator, and the j-th operator in the second branch sequence is determined as the second operator;

[0031] Determine that the first operator and the second operator belong to the set of inter-branch parallel operators.

[0032] In the above scheme, a state transition equation is constructed based on dynamic programming, satisfying the following relationship: the fifth hidden duration is determined by solving this state transition equation based on the first, second, third, and fourth hidden durations. The maximum duration of the fifth hidden duration is taken as the optimal path of the dynamic programming. The i-th operator in the first branch sequence is determined as the first operator, and the j-th operator in the second branch sequence is determined as the second operator.

[0033] In one possible implementation of the first aspect, the fourth hiding duration is 0 when the operator types of the i-th operator and the j-th operator are the same.

[0034] In the above scheme, communication cannot be hidden for operators of the same type from different branch sequences, so the fourth hiding time is 0.

[0035] In one possible implementation of the first aspect, the parallel execution of the first operator and the second operator includes:

[0036] A first control edge is added between the first operator and the second operator according to the set of inter-branch parallel operators;

[0037] The first operator and the second operator are scheduled to complete parallel execution according to the first control edge.

[0038] In the above scheme, the set of parallel operators between branches includes a first operator from the first branch sequence and a second operator from the second branch sequence. A first control edge is added between the first and second operators, and the first and second operators are scheduled to complete parallel execution through the first control edge. It can be understood that control edges can be added between the input data for parallel execution in this embodiment. By controlling the operator scheduling through the first control edge, the first and second operators can be executed in parallel, thereby realizing the parallel execution control of the first and second operators.

[0039] In one possible implementation of the first aspect, the method further includes:

[0040] The results of the operator execution are merged by merging operators inserted into the target model to obtain a merged result.

[0041] In the above scheme, a merging operator is inserted into the target model. This merging operator can be used to merge the results of operator execution to obtain a merged result, thereby reducing computation time and storage space and improving the computational efficiency of the model.

[0042] In one possible implementation of the first aspect, the segmentation of the input data in the target model includes:

[0043] The input data in the target model is segmented by a segmentation operator inserted into the target model.

[0044] In the above scheme, a segmentation operator is inserted into the target model to segment the input data in the target model into two segmented data sets. In this embodiment, the segmentation operator inserted into the target model can complete the task of segmenting the input data.

[0045] Secondly, embodiments of this application also provide a parallel processing apparatus, comprising:

[0046] The segmentation module is used to segment the input data in the target model to obtain the first segmented data set and the second slice data set;

[0047] The data processing module is used to process the first segmented data set and the second sliced ​​data set respectively through the parallel network module to obtain a first branch sequence and a second branch sequence, wherein the first branch sequence and the branch sequence respectively include multiple operators;

[0048] The inter-branch operator merging module is used to determine a set of inter-branch parallel operators from the first branch sequence and the second branch sequence through dynamic programming. The set of inter-branch parallel operators includes: a first operator from the first branch sequence and a second operator from the second branch sequence.

[0049] The parallel execution module is used to execute the first operator and the second operator in parallel to obtain the operator execution result.

[0050] In the above scheme,

[0051] In a second aspect of this application, the constituent modules of the parallel processing apparatus may also perform the steps described in the first aspect and various possible implementations, as detailed in the foregoing description of the first aspect and various possible implementations.

[0052] Thirdly, embodiments of this application provide a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the method described in the first aspect above.

[0053] Fourthly, embodiments of this application provide a computer program product containing instructions that, when run on a computer, cause the computer to perform the method described in the first aspect above.

[0054] Fifthly, embodiments of this application provide a communication device, which may include entities such as terminal devices or chips. The communication device includes: a processor and a memory; the memory is used to store instructions; the processor is used to execute the instructions in the memory, causing the communication device to perform the method as described in any one of the preceding first aspects.

[0055] Sixthly, this application provides a chip system including a processor for supporting a parallel processing device in implementing the functions involved in the foregoing aspects, such as transmitting or processing data and / or information involved in the foregoing methods. In one possible design, the chip system further includes a memory for storing program instructions and data necessary for the parallel processing device. This chip system may be composed of chips or may include chips and other discrete devices.

[0056] In a seventh aspect, embodiments of this application provide a chip including one or more interface circuits and one or more processors; the interface circuits are used to receive signals from the memory of an electronic device and send signals to the processors, the signals including computer instructions stored in the memory; when the processor executes the computer instructions, it causes the electronic device to perform the parallel processing method in the first aspect or any possible implementation of the first aspect.

[0057] The seventh aspect and any implementation thereof correspond to the first aspect and any implementation thereof, respectively. The technical effects of the seventh aspect and any implementation thereof are similar to those of the first aspect and any implementation thereof, and will not be repeated here. Attached Figure Description

[0058] Figure 1 This is a schematic diagram of a data center architecture provided in an embodiment of this application;

[0059] Figure 2a This is an exemplary system diagram provided in an embodiment of this application;

[0060] Figure 2b This is another exemplary system diagram provided in the embodiments of this application;

[0061] Figure 3 This is a schematic diagram of an embodiment of a parallel processing method provided in this application;

[0062] Figure 4 This is a schematic diagram of an input operator in a model provided in an embodiment of this application;

[0063] Figure 5 This is a schematic diagram of the system architecture used in a parallel processing method provided in an embodiment of this application;

[0064] Figure 6 This is a schematic diagram of a method for segmenting input data according to an embodiment of this application;

[0065] Figure 7 This is a schematic diagram illustrating the propagation of branch identifiers in a branch sequence according to an embodiment of this application;

[0066] Figure 8 This is a schematic diagram illustrating the runtime of an evaluation operator provided in an embodiment of this application;

[0067] Figure 9 This is a schematic diagram of a method for selecting the optimal parallel operator based on dynamic programming, provided in an embodiment of this application.

[0068] Figure 10 This is a schematic diagram illustrating the operator runtime in a first branch sequence and a second branch sequence according to an embodiment of this application;

[0069] Figure 11 This is a schematic diagram illustrating the addition of control edges based on the optimal path of dynamic programming, provided in an embodiment of this application.

[0070] Figure 12 This is a schematic diagram illustrating the merging and execution of two branch sequences according to a control edge, provided in an embodiment of this application.

[0071] Figure 13 This is a schematic diagram of an embodiment of a parallel processing device provided in this application;

[0072] Figure 14This is a schematic diagram of the structure of a computing device provided in an embodiment of this application;

[0073] Figure 15 This is a schematic diagram of a computing device cluster provided in an embodiment of this application;

[0074] Figure 16 This is a schematic diagram of a computing device cluster provided in an embodiment of this application. Detailed Implementation

[0075] This application provides a parallel processing method, parallel processing device, electronic device, and storage medium to improve model computation efficiency and increase processor hardware utilization.

[0076] In this application embodiment, the number of nouns, unless otherwise specified, refers to "singular nouns or plural nouns," that is, "one or more." "At least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, or B exists alone, where A / B can be singular or plural. The character " / " generally indicates that the related objects before and after are in an "or" relationship. For example, A / B means: A or B. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, and / or c means the following combinations: a exists alone, b exists alone, c exists alone, a and b exist simultaneously, a and c exist simultaneously, b and c exist simultaneously, or a, b, and c exist simultaneously, where a, b, and c can be single or multiple.

[0077] In the embodiments of this application, "when," "if," and "if" all refer to the device taking corresponding actions under certain objective circumstances, and are not time-limited, nor do they require the device to perform a judgment action, nor do they imply any other limitations. Unless otherwise specified, "if" and "if" can be substituted, and "when" and "in the case of" can be substituted. "When" and "if" / "if" can be substituted.

[0078] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0079] In this application, the ordinal numbers such as "first" and "second" are used to distinguish multiple objects, and are not used to limit the size, content, order, timing, priority, or importance of the multiple objects. For example, the first beam set and the second beam set refer to two different beam sets, and do not indicate that the two beam sets have different priorities or importance.

[0080] This application provides a parallel processing method that can be applied to a computing device cluster, which may include one or more computing devices.

[0081] The type of computing device is not limited here. For example, any computing device can be a terminal device, a cloud server, a container, or a virtual machine, etc.

[0082] The computing device cluster can provide a data processing platform to implement the parallel processing method of the embodiments of this application. Specifically, a data processing platform can be provided in the terminal device and in the cloud server. Based on the respective data processing platforms of the terminal device or the cloud server, consistency and computing accuracy between the terminal side and the cloud side can be achieved, while also making full use of the differences in computing power between different devices. The specific form of the computing device cluster and the corresponding data processing platform is not limited here.

[0083] In one example, the cluster of computing devices can be used to implement a cloud management platform; in other words, the data processing platform of this application embodiment can be implemented through a cloud management platform.

[0084] A cloud management platform is used to manage the infrastructure that provides cloud services. It can provide computing, networking, and storage capabilities based on hardware and software resources. For example, a cloud management platform may include one or more data centers to provide cloud resources through one or more data centers.

[0085] The following is combined Figure 1 The diagram shown illustrates an architecture for a data center.

[0086] Figure 1 In this data center, the cloud management platform communicates with one or more servers (such as...) via the data center's internal network. Figure 1Servers 1 and 2 interact with each other. The server comprises a hardware layer and a software layer. The hardware layer includes the server's hardware configuration, such as peripheral component interconnect (PCI) devices (e.g., network cards, graphics processing units, GPUs, offloading cards, etc.) that can be plugged into peripheral component interconnect (PCI) or peripheral component interconnect express (PCIe) slots. The software layer includes the operating system installed and running on the server (the operating system relative to the virtual machine can be called the host operating system). The host operating system contains a virtual machine manager (also called a hypervisor), whose role is to implement computational virtualization, network virtualization, and storage virtualization of the virtual machine, and to manage the virtual machine. A virtual machine (virtual machine) refers to a complete computer system simulated by software, possessing full hardware system functionality, and running in a completely isolated environment. Figure 1 In the system architecture shown, a data center contains multiple servers, which can be used to run virtual machines. The specifications of the virtual machines can be the same or different. These virtual machines can also be called Elastic Compute Service (ECS), Elastic Instances, etc., and different cloud service providers may use different names.

[0087] In one example of the embodiments of this application, the cloud management platform can be a public cloud platform. In this case, cloud service providers such as individuals or software developers with cloud resource development capabilities can provide cloud services to users. Users obtain cloud services through the Internet, but do not own cloud computing resources.

[0088] Specifically, in Figure 1 In the example shown, the cloud management platform can provide an access interface (such as a user interface or application programming interface (API)). Users of the cloud management platform and cloud service providers can operate the client to remotely access the access interface to register a cloud account and password on the cloud management platform. After the cloud management platform successfully authenticates the cloud account and password, they can log in to the cloud management platform to create, manage, log in to and operate virtual machines in the cloud data center, thereby performing corresponding tasks through the cloud resources of the cloud management platform.

[0089] For example, such as Figure 2aIn the example shown, the data processing platform that implements the parallel processing method in the embodiments of this application can be provided to users in the form of cloud services.

[0090] When data processing tasks require cross-device storage, some enterprises, organizations, or individuals can purchase cloud services that include the data processing platform. They can then use the cloud resources of the cloud management platform to perform the relevant data processing tasks and obtain the processing results of the input data from the cloud management platform.

[0091] This data processing platform can be implemented based on cloud resources such as one or more cloud data centers in the cloud management platform.

[0092] This data processing platform can be provided to users as an independent cloud service, and it is also compatible with computing frameworks such as TensorFlow and PyTorch. In other words, it can be used as an operator in computing frameworks such as TensorFlow and PyTorch, providing functions such as preprocessing, computation, feature extraction and / or other data processing for real-time and offline data.

[0093] Of course, the cloud management platform can also be other types of cloud management platforms, and this application embodiment does not limit this.

[0094] In another example, such as Figure 2b As shown, the data processing platform can be provided to the user in the form of a software product or a functional service within a software product. The software product can be deployed on the user's terminal device or on the enterprise user's server cluster. The user can purchase a software license to obtain the right to use the software product and, within the license period, implement the parallel processing method of the embodiments of this application through the software product.

[0095] The following example, using the field of parallel processing methods, illustrates the practical application of a data processing platform that implements this parallel processing method. This data processing platform has the functionality of the parallel processing device described in subsequent embodiments. Specifically, this parallel processing device can be a terminal device or a server. For example, the terminal device or server includes a processor, which is used to execute the parallel processing method described in the first aspect above.

[0096] For example, a standard process may specifically include computer program code, and a parallel processing device may specifically be a code executor.

[0097] For example, a parallel processing device can specifically be an agent that incorporates a large model. The agent can be used to execute the parallel processing methods provided in the first aspect above.

[0098] Within the framework of intelligent agents, agents are endowed with the general capabilities of large models to solve one or more specific problems. Different types of intelligent agents can be automatically linked together to solve more complex problems.

[0099] In one exemplary application scenario, a user might want to transmit PCB layout defect information to a cloud server for storage, sharing, or further processing. The user uses computing frameworks such as TensorFlow and PyTorch to encode the PCB layout defect information using an AI model, and then sends the encoded bitstream via a transmission network between the mobile phone and the cloud server. This computing framework can be deployed on the client device or on the cloud server. After receiving the compressed PCB layout defect information, the cloud management platform utilizes cloud resources to process the information. For example, the cloud management platform might include a large language model, a detection tool library, and a PCB layout engine. The platform performs semantic understanding of the PCB layout defect information to obtain a defect description, uses a target detection tool to detect defects in the description, and derives optimization instructions based on the detected electronic components on the target PCB. Finally, the platform optimizes the layout of these electronic components based on these optimization instructions.

[0100] It is understood that the above example is only an exemplary introduction to one application scenario, and the parallel processing method of this application embodiment can be applied to multiple fields, and is not limited here.

[0101] Currently, large-scale models can employ a Mixture of Experts (MoE) architecture, providing a method to increase the number of model parameters without increasing computational costs. Large models based on the MoE architecture are sparse large models using conditional computation, becoming a mainstream solution for improving model capacity while maintaining computational efficiency. This involves selecting only a subset of model parameters for activation through sparse conditional computation strategies, thereby increasing model capacity and performance without proportionally increasing computational cost. Mainstream large models such as Mixtral, Pangu, DeepSeek, and Qwen all incorporate this structure.

[0102] Large models in the MoE architecture typically have a large number of parameters, requiring additional expert parallelism for model partitioning. To achieve expert parallelism during the training of large MoE models, different experts need to be assigned to multiple devices. The expert parallelism strategy introduces a time-consuming all-to-all (A2A) communication operator. The specific process of all-to-all communication is as follows:

[0103] First, input allocation is performed. The input data (token) on each device is routed to the corresponding expert through a gating function, generating the input data that each expert needs to process.

[0104] Next, All-to-All communication is performed. Input data from each device needs to be transmitted to experts on other devices, which triggers All-to-All communication. In All-to-All communication, each operation sends the input data belonging to experts on other devices from each device to the corresponding target device via the network.

[0105] Next, expert calculations are performed. After receiving the input, the experts on each device begin to perform their forward or backward calculations.

[0106] Finally, the results are returned. After the calculation is completed, the output of each expert is sent back to the original device again via All-to-All communication for further aggregation and subsequent calculations.

[0107] As the cluster size increases, the proportion of A2A communication also increases, becoming a bottleneck for model computation. In expert parallel workflows, all-to-all communication overhead is significant, making it a key area for optimization.

[0108] Currently, there is an optimization scheme using a group-wise All-to-All process. This scheme overcomes the drawback of the original All-to-All communication, which requires a large amount of data transmission between nodes. Instead, it uses partial data communication between nodes plus partial data communication within each node to achieve the same effect, effectively improving communication efficiency. Specifically, the group-wise All-to-All only performs group-to-group All-to-All communication between nodes. Within each node, the AllGather operator is used to distribute the input data (token) to the processor within each node, saving the communication overhead between nodes.

[0109] The above scheme employs a more efficient communication protocol and network, reducing all-to-all communication overhead and alleviating the problem of high model computation to some extent. However, the addition of the intra-node AllGather operator still results in significant time consumption, leading to low model computation efficiency and low processor hardware utilization.

[0110] To address the issues of low computational efficiency and low processor hardware utilization in current large models, this application provides a parallel processing method and a corresponding parallel processing device. The parallel processing method divides the input data of a large model into two branch sequences, determines parallel operators between branches through dynamic programming, and executes these parallel operators in parallel. This method is suitable for large models with a large number of parameters in the MoE architecture. It optimizes communication overhead without affecting model performance, improves the training efficiency of large models, and enhances processor utilization, such as improving the utilization rate of the Neural Processing Unit (NPU).

[0111] like Figure 3 As shown, a parallel processing method executed by a parallel processing device may include the following steps 301-304.

[0112] 301. The input data in the target model is segmented to obtain the first segmented data set and the second slice data set.

[0113] In this embodiment, the target model can be a machine learning model. For example, the target model can be a large model of the MoE architecture with a large number of model parameters. Any model that can achieve parallel processing can be used as the target model in this embodiment, and there is no limitation here.

[0114] The input data in the target model in this embodiment may include multiple text processing units (tokens). Taking M as an example, the number of operators input into the target model is M. The value of M is not limited in this embodiment. In natural language processing, a token refers to the smallest semantic unit in text. A token can be called a text processing unit. For example, a sentence can be divided into several words, and each word is a token.

[0115] In this embodiment, the target model includes input data. The input data is segmented to obtain a first segmented data set and a second sliced ​​data set. For example, if the input data in the target model is a 512x512 data matrix, segmenting this input data can divide it into two sliced ​​data sets, each of which is 256x256. This embodiment does not limit the segmentation method of the input data in the target model.

[0116] It is understood that, in step 301 of this application embodiment, the input data is divided into two segmented data sets as an example. It is not limited to this. In this application embodiment, the input data input to the target model can also be divided into three or more segmented data sets. For multiple segmented data sets, subsequent steps 302 and 304 can also be executed to generate multiple branch sequences and identify the operators that can be executed in parallel in the multiple branch sequences.

[0117] This application embodiment obtains a first segmented data set and a second sliced ​​data set by segmenting the input data. Multiple text processing units in each segmented data set can be assigned to multiple experts, who can be deployed on a computing card. Operators running on the computing card can then process the text processing units.

[0118] 302. The first segmented data set and the second sliced ​​data set are processed by a parallel network module to obtain a first branch sequence and a second branch sequence, wherein the first branch sequence and the branch sequence each include multiple operators.

[0119] In this embodiment, after obtaining the first segmented data set and the second slice data set, a parallel network module is used to process the first slice data set to obtain the first branch sequence, and the same parallel network module is used to process the second slice data set to obtain the second branch sequence. Specifically, in this embodiment, the parallel network module processes the slice data set by using a gating function to route the slice data in the slice data set to the corresponding experts, with each expert processing the slice data in their respective slice data set. In this embodiment, the parallel network module refers to a network module that supports expert parallelism. For example, this parallel network module could be a module of a large model within the MoE architecture. The parallel network module allows multiple devices to compute in parallel, thereby improving the efficiency of model training and inference.

[0120] In this embodiment, a parallel network module processes the first segmented data set and the second sliced ​​data set respectively to obtain a first branch sequence and a second branch sequence. Each branch sequence includes multiple operators, which are operators that can perform operations on variables, data structures, or expressions. For example, operators are functions in mathematics, operators in algebra, or functions or methods in programming languages. Operators can change the input value or data structure and produce output results. For example, in this embodiment, multiple intermediate representations (IRs) input into the target model are segmented to obtain at least two segmented data sets. The parallel network module processes these at least two segmented data sets, for example, during the static graph compilation stage. After iteratively expanding the at least two segmented data sets, the parallel network module can obtain at least one branch sequence, each including multiple operators. In this embodiment, the multiple operators can be categorized into communication operators and computation operators. Figure 4 As shown, a communication operator refers to an operator used for communication between operators. For example, a communication operator may include at least one of the following: a fully exchanged data communication operator (All2All), a fully aggregated data communication operator (AllGather), and a scattering reduction operator (Reducescatter). This application embodiment does not limit the type of communication operator. A computation operator refers to an operator that performs calculations within itself. For example, a computation operator may include at least one of the following: a multiplication operator (Multiply), an addition operator (Add), and a branch matrix multiplication operator (BatchMatMul). This application embodiment does not limit the type of computation operator.

[0121] In this embodiment, the first segmented data set and the second sliced ​​data set are processed by a parallel network module to form two branch sequences that can be executed in parallel. Since the operator operation and communication time in the target model are proportional to the amount of data, as the amount of data in the input branch sequence is halved, the operation and communication time of each branch may be halved. In the case that operators are executed in parallel in the two branch sequences, communication masking can be achieved.

[0122] It is understood that, in step 301 of this application embodiment, the input data is divided into two segmented data sets as an example. It is not limited to this. In this application embodiment, the input data input to the target model can also be divided into three or more segmented data sets. For multiple segmented data sets, multiple segmented data sets are processed by a parallel network module, and then subsequent steps 303 and 304 can be executed to identify the operators that can be executed in parallel in multiple branch sequences.

[0123] In some embodiments of this application, in addition to performing the aforementioned steps 301 and 302, the method performed by the parallel processing device may also include the following steps:

[0124] A1. Remove the operators that are visited together from the first branch sequence and the second branch sequence to obtain the first branch sequence and the second branch sequence after removing the operators.

[0125] In the scenario of executing step A1, step 303 above determines the set of parallel operators between branches from the first branch sequence and the second branch sequence through dynamic programming, including:

[0126] A2. Determine the set of parallel operators between branches from the first branch sequence after removing operators and the second branch sequence after removing operators using dynamic programming.

[0127] In this embodiment, operators commonly accessed in both the first and second branch sequences refer to operators with dependencies between them. By analyzing the dependencies between different branch sequences, commonly accessed operators are identified; these commonly accessed operators are those that cannot be executed in parallel in both the first and second branch sequences. After identifying the commonly accessed operators, these operators are removed from both sequences to obtain a first branch sequence and a second branch sequence after operator removal. Dynamic programming is then used to determine the set of parallel operators between branches from these two sequences. In this embodiment, by removing commonly accessed operators from both branch sequences, dependent operators are eliminated, ensuring that there are no dependencies when operators are executed in parallel between the two branch sequences, thus avoiding runtime errors in the model.

[0128] In some embodiments of this application, step 301 involves segmenting the input data in the target model, including:

[0129] B1. The input data in the target model is segmented by a segmentation operator inserted into the target model.

[0130] In this embodiment, a splitting operator is inserted into the target model. A splitting operator, also known as a segmentation operator, is an operator used to cut input data according to specified rules. In this embodiment, a splitting operator is inserted into the target model to split the input data into two split data sets. The splitting operator inserted into the target model in this embodiment can complete the task of splitting the input data.

[0131] Furthermore, in some embodiments of this application, step A1, which removes commonly accessed operators from the first branch sequence and the second branch sequence, includes:

[0132] A11. Starting from the merge operator inserted into the target model, branch label propagation is performed on the first branch sequence and the second branch sequence, and branch label propagation is stopped at the splitting operator inserted into the target model.

[0133] This process involves inserting a segmentation operator and a concat operator into the target model. The concat operator merges the operators within the target model. These segmentation and concat operators are computational operators. Starting with the concat operator, branch labels (IDs) are propagated for the first and second branch sequences. Branch label propagation stops at the segmentation operator. For example, starting with the concat operator, label propagation proceeds in reverse order for the first and second branch sequences. Operators commonly accessed by both sequences are not propagated; that is, commonly accessed operators are removed from both sequences. Branch label propagation stops at the segmentation operator.

[0134] 303. Determine the set of parallel operators between branches from the first branch sequence and the second branch sequence by dynamic programming. The set of parallel operators between branches includes: the first operator from the first branch sequence and the second operator from the second branch sequence.

[0135] In this embodiment, after determining the two branch sequences, a dynamic programming (DP) algorithm can be used to construct the state transition equation, thereby determining the set of parallel operators between branches. This set of parallel operators represents the set of operators that can be executed in parallel in the two branch sequences. In this embodiment, the set of parallel operators between branches includes a group of operators that can be executed in parallel from the first branch sequence and the second branch sequence. The parallel execution of operators in the set of parallel operators between branches enables communication masking.

[0136] The dynamic programming approach used in this embodiment is a process of optimizing the decision-making process. The state transition equation in dynamic programming is the key equation used to describe state changes, representing the transition from operators in one branch sequence to operators in another. By solving the state transition equation, the state of the next stage can be determined from the current state and the decision. In this embodiment, the dynamic programming can be represented in functional form, ensuring the continuity of states and the effectiveness of decisions.

[0137] In this embodiment, a set of inter-branch parallel operators determined by dynamic programming is used. Operators from different branch sequences within this set of inter-branch parallel operators can perform communication overlapping. That is, by slicing and grouping the input data to be processed on the device or processor, communication operations in another branch sequence are executed while the computational operation of one branch sequence is being performed, thereby achieving communication masking and improving the model FLOPs utilization (MFU).

[0138] In this embodiment, dynamic programming is used to determine a first operator from a first branch sequence and a second operator from a second branch sequence. The first and second operators are two different types of operators, and both are operators that can be executed in parallel. The set of parallel inter-branch operators generated in this embodiment includes the first and second operators. In this embodiment, the first operator is the operator in the first branch sequence that can be executed in parallel with the second operator, and the second operator is the operator in the second branch sequence that can be executed in parallel with the first operator. Not limited in scope, the number of operators included in the inter-branch parallel operator set in this embodiment can be multiple. For example, the inter-branch parallel operator set may include three first operators from the first branch sequence: first operator 1, first operator 2, and first operator 3; and the inter-branch parallel operator set may include three second operators from the second branch sequence: second operator 1, second operator 2, and second operator 3. In this case, first operator 1 and second operator 1 are operators that can be executed in parallel; first operator 2 and second operator 2 are operators that can be executed in parallel; and first operator 3 and second operator 3 are operators that can be executed in parallel. In this embodiment, the purpose of communication hiding can be achieved through the parallel execution of first operator 1 and second operator 1, the parallel execution of first operator 2 and second operator 2, and the parallel execution of first operator 3 and second operator 3.

[0139] In some embodiments of this application, the parallel processing method executed by the parallel processing device further includes:

[0140] C1. Merge consecutive operators of the same type within the first and second branch sequences to obtain the merged first branch sequence and the merged second branch sequence.

[0141] Specifically, for the first and second branch sequences, consecutive operators of the same type within each sequence can be merged to obtain the merged first and second branch sequences. For example, consecutive operators refer to those executed sequentially within the branch sequence, and operators of the same type refer to those with the same operator type. For instance, if two operators are both communication operators, then their types are the same; or if two operators are both computation operators, then their types are the same. By merging consecutive operators of the same type, the execution efficiency of operators in the target model can be improved.

[0142] In some embodiments of this application, in the implementation scenario where the parallel processing device executes C1, step 303 determines the set of inter-branch parallel operators from the first branch sequence and the second branch sequence through dynamic programming, including:

[0143] D1. Determine the set of parallel operators between branches from the first branch sequence after the merging operator and the second branch sequence after the merging operator using dynamic programming.

[0144] In the first branch sequence, operators within the sequence are merged, and in the second branch sequence, operators within the sequence are merged. Next, dynamic programming can be performed on the first branch sequence after merging operators and the second branch sequence after merging operators. Since dynamic programming is performed on the two branch sequences after merging operators, the efficiency of generating a set of parallel operators between branches can be improved.

[0145] In some embodiments of this application, step C1 merges consecutive operators of the same type within the first branch sequence and the second branch sequence, including:

[0146] C11. Get the runtime of the operators in the first branch sequence and the second branch sequence.

[0147] In this embodiment, the execution of each operator within the first and second branch sequences will consume a certain runtime, which can also be referred to as execution time. This embodiment does not limit the method of obtaining the operator's runtime. For example, it can look up the runtime of each operator based on a fixed operator execution time table, or perform performance analysis (profiling) on ​​each operator within the first and second branch sequences to determine the runtime of each operator, or determine the runtime of each operator based on the user's configuration information for the operators in the target model.

[0148] C12. Obtain the sum of runtime for consecutive operators of the same type within the first and second branch sequences.

[0149] Specifically, for consecutive operators of the same type in each branch sequence, the sum of the runtimes of consecutive operators of the same type in the branch sequence is calculated. For example, if there are 3 consecutive operators of the same type in the first branch sequence, the sum of the runtimes of these 3 operators is obtained based on the runtime of each of these 3 operators.

[0150] C13. If the total runtime is less than the first runtime threshold, merge consecutive operators of the same type within the first branch sequence and the second branch sequence.

[0151] Whether to merge consecutive operators of the same type within each sequence can be determined by whether the total runtime is less than a first time threshold. If the total runtime is less than the first time threshold, operators can be merged within their respective sequences, thereby improving the operator running efficiency in the target model. If the total runtime of consecutive operators of the same type within a branch sequence exceeds the first time threshold, only consecutive operators of the same type within the branch sequence whose total runtime is less than the first time threshold are merged. For example, if there are four consecutive operators of the same type in the first branch sequence, and the total runtime of these four operators is greater than or equal to the first time threshold, while the total runtime of the first three operators is less than the first time threshold, only the first three operators are merged. The specific value of the first time threshold is not limited in this embodiment. In this embodiment of the application, when the sum of the runtime of consecutive operators of the same type in the first branch sequence and the second branch sequence is less than the first runtime threshold, consecutive operators of the same type in the first branch sequence and the second branch sequence are merged to avoid operators with excessive runtime after merging, thereby improving the operator execution efficiency in the target model.

[0152] In some embodiments of this application, the dynamic programming process is illustrated by considering whether the i-th operator in the first branch sequence and the j-th operator in the second branch sequence can be used as parallel inter-branch operators. The values ​​of i and j are not limited; for example, i = 1 represents the first operator in the first branch sequence, and j = 1 represents the first operator in the second branch sequence. In the embodiments of this application, the hidden time for communication and computation between the i-th and j-th operators is related to the hidden time for communication and computation between the (i-1) operators preceding the i-th operator in the first branch sequence and the (j-1) operators preceding the j-th operator in the second branch sequence. Here, the hidden time refers to the time hidden when operators in the two branch sequences are executed in parallel.

[0153] Step 303 determines the set of inter-branch parallel operators from the first branch sequence and the second branch sequence using dynamic programming, including:

[0154] E1. Obtain the first hidden duration for communication and computation between the first (i-1) operators in the first branch sequence and the first j operators in the second branch sequence, where i and j are positive integers.

[0155] Here, the first (i-1) operators in the first branch sequence refer to the first to (i-1)th operators in the first branch sequence, and the first j operators in the second branch sequence refer to the first to jth operators in the second branch sequence. The first hiding time refers to the time that can be hidden when the first (i-1) operators in the first branch sequence communicate and compute with the first j operators in the second branch sequence.

[0156] E2. Obtain the second hidden duration for communication and computation between the first i operators in the first branch sequence and the first (j-1) operators in the second branch sequence.

[0157] Here, the first i operators in the first branch sequence refer to the first to the i-th operators in the first branch sequence, and the first (j-1) operators in the second branch sequence refer to the first to the (j-1)-th operators in the second branch sequence. The second hiding time refers to the time that can be hidden when the first i operators in the first branch sequence communicate and compute with the first (j-1) operators in the second branch sequence.

[0158] E3. Obtain the third hidden duration for communication and computation between the first (i-1) operators in the first branch sequence and the first (j-1) operators in the second branch sequence, and the fourth hidden duration for communication and computation between the i-th operator in the first branch sequence and the j-th operator in the second branch sequence.

[0159] Here, the first (i-1) operators in the first branch sequence refer to the first to (i-1)th operators in the first branch sequence, and the first (j-1) operators in the second branch sequence refer to the first to (j-1)th operators in the second branch sequence. The third hiding time refers to the time that can be hidden when the first (i-1) operators in the first branch sequence and the first (j-1) operators in the second branch sequence communicate and compute.

[0160] In addition, for the i-th operator in the first branch sequence and the j-th operator in the second branch sequence, the hidden duration when the i-th operator and the j-th operator communicate and compute directly is used as the fourth hidden duration.

[0161] E4. Obtain the fifth hidden duration based on the first hidden duration, the second hidden duration, the third hidden duration, and the fourth hidden duration. The fifth hidden duration is the hidden duration for communication and computation between the first i operators in the first branch sequence and the first j operators in the second branch sequence.

[0162] Wherein, the first hidden duration represents the hidden duration for communication and computation between the first (i-1) operators in the first branch sequence and the first j operators in the second branch sequence; the second hidden duration represents the hidden duration for communication and computation between the first i operators in the first branch sequence and the first (j-1) operators in the second branch sequence; the result of adding the third and fourth hidden durations can be used as the hidden duration for communication and computation starting from the first operator in the first branch sequence and the first operator in the second branch sequence, up to the i-th and j-th operators. In this embodiment, the fifth hidden duration is the hidden duration for communication and computation between the first i operators in the first branch sequence and the first j operators in the second branch sequence. Based on dynamic programming, a state transition equation is constructed, satisfying the following relationship: the fifth hidden duration can be determined by solving the state transition equation based on the first, second, third, and fourth hidden durations.

[0163] E5. Using the fifth hidden duration as the maximum duration as the optimal path for dynamic programming, determine the i-th operator in the first branch sequence as the first operator and the j-th operator in the second branch sequence as the second operator.

[0164] In this process, the maximum value of the fifth hidden duration is selected as the optimal path for dynamic programming. That is, communication and computation begin from the first operator in the first branch sequence and the first operator in the second branch sequence, until the hidden duration of communication and computation of the i-th operator and the j-th operator reaches the maximum value. At this point, the i-th operator and the j-th operator are determined to be a set of operators that can be executed in parallel in the first branch sequence and the second branch sequence. The i-th operator is the first operator determined in step 303, and the j-th operator is the second operator determined in step 303.

[0165] E6. Determine if the first and second operators belong to the set of inter-branch parallel operators.

[0166] In step E5, the first operator and the second operator are determined, and a set of parallel operators between branches is generated based on the first operator and the second operator.

[0167] It is understood that embodiments E1 to E6 of this application only describe the determination method of a first operator and a second operator. The values ​​of i and j are not limited in this application embodiment. The configuration of i and j according to the actual scenario can determine whether other operators in the first branch sequence and the second branch sequence can be executed in parallel. This will not be elaborated here.

[0168] Furthermore, in some embodiments of this application, when the operator types of the i-th operator and the j-th operator are the same, the fourth hiding duration is 0.

[0169] In this embodiment, the i-th operator comes from the first branch sequence, and the j-th operator comes from the second branch sequence. When operators of the same type from different branch sequences are present, communication hiding is impossible. Therefore, when the i-th and j-th operators have the same operator type, the fourth hiding duration is 0. In this embodiment, communication hiding is impossible for operators of the same type from different branch sequences; therefore, the fourth hiding duration is 0.

[0170] 304. Execute the first and second operators in parallel to obtain the operator execution results.

[0171] In this embodiment, step 303 determines a set of parallel operators between branches. This set includes a first operator and a second operator. The first and second operators are executed in parallel to obtain the operator execution result. In this embodiment, the input data is divided into two branch sequences that can be executed in parallel. Since most operators between the branch sequences do not have dependencies, the first and second operators in the two branch sequences, which have no dependencies, are scheduled in parallel, improving the processor's computational utilization and reducing the training and inference time of the target model.

[0172] In some embodiments of this application, step 304 executes the first operator and the second operator in parallel, including:

[0173] F1. Add a first control edge between the first and second operators based on the set of parallel operators between branches;

[0174] F2. The first operator and the second operator are scheduled according to the first control edge to complete parallel execution.

[0175] The set of parallel operators between branches includes a first operator from the first branch sequence and a second operator from the second branch sequence. A first control edge is added between the first and second operators, and the first and second operators are scheduled to complete parallel execution through the first control edge. It is understood that in this embodiment, control edges can be added between the input data for parallel execution. The first control edge controls the parallel execution of the first and second operators during operator scheduling, thereby achieving parallel execution control of the first and second operators.

[0176] In some embodiments of this application, in addition to performing the aforementioned steps 301 to 304, the method performed by the parallel processing device may also include the following steps:

[0177] G1. Merge the results of operator execution by inserting a merge operator into the target model to obtain the merged result.

[0178] Among them, a merging operator is inserted into the target model. This merging operator can be used to merge the results of operator execution to obtain a merged result, thereby reducing computation time and storage space and improving the computational efficiency of the model.

[0179] As illustrated by the examples in steps 301 to 304 above, in this embodiment of the application, the input data in the target model is segmented to obtain a first segmented data set and a second slice data set. The first segmented data set and the second slice data set are processed by a parallel network module to form two branch sequences that can be executed in parallel. Since most operators in the branch sequences do not have dependencies, the first and second operators in the two branch sequences that do not have dependencies are scheduled in parallel to improve the computational utilization of the processor and reduce the time consumed by training and inference of the target model.

[0180] To facilitate a better understanding and implementation of the above-described solutions in the embodiments of this application, specific examples of corresponding application scenarios are provided below.

[0181] The parallel processing apparatus provided in this application embodiment can be included in deep learning platform software. This parallel processing apparatus does not depend on specific hardware devices but can run on general-purpose computers. Such software can be deployed on various server hardware, including but not limited to high-performance servers in data centers, cloud computing platforms, or personal workstations, as long as these devices have the capability to run deep learning frameworks and process large amounts of data. During runtime, the program code of this application embodiment runs on devices capable of parallel computation and communication.

[0182] This application's embodiments are applicable to large-scale distributed deep learning model training, particularly in training and inference scenarios for Natural Language Processing (NLP) models with trillions of parameters in the MoE architecture. This scenario requires expert parallelism and model parallelism to distribute model parameters across different computing cards to accommodate models with trillions of parameters. Distributed large-scale model training based on expert parallelism and model parallelism necessitates communication between computing cards for data exchange. As the cluster and model size increase, the proportion of communication overhead in distributed training grows significantly. This application's embodiments reduce the proportion of communication overhead and improve NPU utilization by partitioning the input data into a parallelizable branch sequence and dynamically programming to select parallel operators for inter-branch computation and communication.

[0183] Figure 5 This is a schematic diagram of the system architecture used in a parallel processing method provided in this application embodiment. The system architecture provided in this application embodiment may include: a branch label propagation module, an operator runtime evaluation module, a parallel operator determination module, and a parallel execution module.

[0184] The branch label propagation module is used to receive the intermediate representation of the model, split the intermediate representation of the model to obtain two slice data sets, process each slice data set through the module of the large model of the MoE architecture to obtain two branch sequences, and propagate the branch ID in each branch sequence to identify parallel branching operators.

[0185] The operator runtime evaluation module is used to evaluate the runtime of each operator in each branch sequence.

[0186] The parallel operator determination module is used for parallel solution of inter-branch communication computation based on dynamic programming to obtain a set of inter-branch parallel operators. Dynamic programming is used to determine the execution order of each communication and computation operator on each branch sequence, ensuring that the overlap time between communication and computation is maximized.

[0187] The parallel execution module is used to add control edges between branches and between operators in the running set according to the optimal path of dynamic programming, and output the intermediate representation of the optimized model.

[0188] In this embodiment, by analyzing and modifying the intermediate representation of the model, the NPU computing utilization is improved, and the training and inference time of large models with the MoE architecture is reduced.

[0189] Next, regarding Figure 5 The system architecture shown illustrates the parallel processing method provided in the embodiments of this application. The specific steps of the embodiments of this application are as follows:

[0190] Step 1: Propagate BranchId in the static graph to identify operators that can be branched in parallel.

[0191] In this embodiment, the operators in each branch sequence can be divided into two types: computational operators and communication operators. The computational operators in this embodiment may include at least one of the following: multiplication operator (Multiply), addition operator (Add), and branch matrix multiplication operator (BatchMatMul). The communication operators in this embodiment may include at least one of the following: All2All, Allgather, and Reducescatter.

[0192] Insert segmentation and merging operators into model modules that require parallelism. These model modules refer to complete computational units composed of a complex number of simple operators, such as attention modules or network modules in the MoE architecture.

[0193] In step one, the input data is split in the Python data model script, the data slices are executed in a distributed manner, and the results are finally merged together, for example, the input data is split in the following code.

[0194]

[0195]

[0196] like Figure 6 The diagram shown is a schematic representation of input data segmentation according to an embodiment of this application. The data input into the target model can be segmented into a first segmented data set and a second segmented data set. The first and second segmented data sets are processed separately by a parallel network module to obtain a first branch sequence and a second branch sequence. During the static graph compilation stage, after the segmented data sets are iterated through, the input branch sequence is traversed starting from the concat operator to remove common inputs, resulting in a branch sequence that can be executed in parallel.

[0197] like Figure 7 The diagram shown is a schematic diagram of branch identifier propagation for a branch sequence provided in an embodiment of this application. Starting from the merging operator (Concat), the branch ID label is propagated. For example, the label is propagated in reverse order. The operator in the first branch sequence is assigned a branch label (branchid) of 1, the operator in the second branch sequence is assigned a branch label of 2, the branch label (branchid) of the operator that is accessed in the same way is set to 0, the operator that is accessed in the same way is removed from the two branch sequences, and the label propagation process stops at the split operator.

[0198] Step 2: Operator runtime estimation (Estimate Cost).

[0199] The runtime of each operator is determined based on a lookup of the operator execution runtime table, or by performance analysis (profiling) and the user's configuration parameters for the target model. Consecutive operators of the same type are logically merged, i.e., they can be combined into a single fusion operator. If the runtime of the merged operator exceeds a fixed threshold, the merging of consecutive operators of the same type is stopped.

[0200] like Figure 8 The diagram shown is a schematic representation of an operator runtime evaluation method provided in this application embodiment. To evaluate the runtime of each branch operator, the runtime of each operator can be obtained through an operator execution runtime table, performance analysis data, or user configuration. For consecutive operators of the same type, such as two consecutive computational operators, if the total runtime is less than a fixed threshold, these operators are logically merged and treated as a single fusion operator to reduce the size of the operators input in subsequent steps.

[0201] Step 3: Solve the problem in parallel using two-branch communication based on dynamic programming.

[0202] In step three, after obtaining the operator type and runtime, the parallel operators between branches are obtained through dynamic programming based on the state transition equation. The state transition equation for dynamic programming is as follows:

[0203] SequenceOverlap[i,j]

[0204] =max(SequenceOverlap[i-1,j],SequenceOverlap[i,j-1],SequenceOverlap[i-1,j-1]+Overlap(i,j))

[0205] Wherein, SequenceOverlap[i,j] represents the hidden time of the maximum direct communication computation between operators 1 to i in branch sequence 1 and operators 1 to j in branch sequence 2. That is, SequenceOverlap[i,j] represents the hidden time of the maximum direct communication computation between the first i operators in branch sequence 1 and the first j operators in branch sequence 2.

[0206] SequenceOverlap[i,j-1] represents the hidden time of the maximum direct communication computation between operators 1 to i in branch sequence 1 and operators 1 to j-1 in branch sequence 2.

[0207] SequenceOverlap[i-1,j] represents the hidden time of the maximum direct communication computation between operators 1 to i-1 in branch sequence 1 and operators 1 to j in branch sequence 2.

[0208] SequenceOverlap[i-1,j-1] represents the hidden time of the maximum direct communication computation between operators 1 to i-1 in branch sequence 1 and operators 1 to j-1 in branch sequence 2.

[0209] Overlap(i,j) represents the communication computation hiding time between operator i in branch sequence 1 and operator j in branch sequence 2. When operators i and j are both computation operators or both are communication operators, the value of Overlap(i,j) is 0.

[0210] like Figure 9 As shown in the diagram, this application provides a schematic diagram of selecting the optimal parallel operator based on dynamic programming. The horizontal axis represents the multiple operators included in branch sequence 1, with i ranging from 1 to 6. The vertical axis represents the multiple operators included in branch sequence 2, with j ranging from 1 to 6. The arrows indicate the path for selecting the operator that can be executed in parallel from the two branch sequences.

[0211] For any i-th operator and j-th operator, the hidden duration of the maximum communication computation, SequenceOverlap[i,j], can be calculated according to the above state transition equation.

[0212] The specific algorithm flow is as follows:

[0213]

[0214]

[0215] Among them, Figure 9 In the two branch sequences, branch sequence 1 and branch sequence 2 are traversed respectively, and the hidden time when operators are executed in parallel in the upward, leftward, and diagonal directions is calculated. For the hidden time in the diagonal direction, diag_cost = cost_map[i-1][j-1] + overlap. The maximum hidden time in the upward, leftward, and diagonal directions is selected, which is the hidden time calculated by the maximum direct communication between the first i operators in branch sequence 1 and the first j operators in branch sequence 2.

[0216] Step 4: Based on the optimal path derived from dynamic programming, add control edges to operators that can be executed in parallel between branches. These control edges enable the communication and computation operators in different branches to execute in parallel.

[0217] like Figure 10 The diagram shown is a schematic of adding control edges based on the optimal path of dynamic programming according to an embodiment of this application. Step four: After obtaining the optimal path of dynamic programming, for the parallelizable operators of the two branch sequences on the path, control edges are inserted to enable the computation operator and the communication operator to be executed in parallel during the scheduling of the control operator.

[0218] Specific algorithm steps:

[0219]

[0220] Specifically, based on the optimal path of dynamic programming, branch1_nodes[lhs_index] from branch sequence 1 and branch2_nodes[rhs_index] from branch sequence 2 are determined, control edges are established, and parallelizable operators are scheduled and executed through the control edges.

[0221] Step 5: Merge the branch results.

[0222] like Figure 11 The diagram shows the operator runtime in the first and second branch sequences. Figure 12The diagram shown is a schematic of merging two branch sequences according to the control edge provided in an embodiment of this application. The results of each branch are merged on the computation graph by the merging operator (concat).

[0223] In this embodiment, the data entering the MoE architecture model is split and inter-branch communication computations are performed in parallel. During the original training process, computation operators and communication operators wait for each other, with communication operators consuming significant time and resulting in low NPU utilization. Splitting the input data into two halves creates two computational branches. Since operator computation and communication time in the network are proportional to the amount of data, as the data volume of each input branch is halved, the computation and communication time of each branch is almost entirely halved. Finally, the output results of each branch are merged. Without inter-branch parallelism, the branches are executed serially after splitting, and the communication computation time after merging is almost identical to that before splitting. Considering that most operators between branches do not have dependencies, control edges are inserted to schedule computation and communication operators without dependencies between the two branches in parallel, improving NPU computational utilization and reducing training runtime.

[0224] by Figure 12 Taking the example shown, the operator runtime for the first branch sequence is 8+1+1+8+8+8=34, and the operator runtime for the second branch sequence is 8+1+1+8+8+8=34. If the first and second branch sequences are run separately, the required operator runtime is 34+34=68. However, according to the parallel processing method provided in this application embodiment, the operator runtime for the two branch sequences is 8+1+1+8+1+8+8+8+8=51, thus improving the operator runtime efficiency of the target model by (68-51) / 68=25%.

[0225] The solution provided in this application is expected to reduce the communication ratio and improve the training performance of the model by more than 20% in the training of NLP models with a trillion-scale MoE architecture by generating parallel branch sequences through data partitioning and then automatically selecting the best parallel operator by dynamic programming.

[0226] After data partitioning, communication and computation operators for different branch sequences are executed in parallel. The globally optimal parallel operator is automatically selected based on dynamic programming, hiding approximately 50% of the module's communication overhead. This dynamic programming-based automatic selection of parallel operators is applicable to all networks, eliminating the need for manual configuration and improving the operational efficiency of operators in the model.

[0227] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0228] To facilitate better implementation of the above-described solutions in the embodiments of this application, related apparatus for implementing the above-described solutions is also provided below.

[0229] Please see Figure 13 As shown in the figure, a parallel processing device 1300 provided in this application embodiment may include: a segmentation module 1301, a data processing module 1302, an inter-branch operator merging module 1303, and a parallel execution module 1304, wherein,

[0230] The segmentation module is used to segment the input data in the target model to obtain the first segmented data set and the second slice data set;

[0231] The data processing module is used to process the first segmented data set and the second sliced ​​data set respectively through the parallel network module to obtain a first branch sequence and a second branch sequence, wherein the first branch sequence and the branch sequence respectively include multiple operators;

[0232] The inter-branch operator merging module is used to determine a set of inter-branch parallel operators from the first branch sequence and the second branch sequence through dynamic programming. The set of inter-branch parallel operators includes: a first operator from the first branch sequence and a second operator from the second branch sequence.

[0233] The parallel execution module is used to execute the first operator and the second operator in parallel to obtain the operator execution result.

[0234] In this embodiment, the module is an example of a software functional unit, and the data processing device may include code running on a computing instance. The computing instance may be at least one of a physical host (computing device), a virtual machine, a container, or other computing devices. Further, the aforementioned computing device may be one or more. For example, the data processing device may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the application may be distributed in the same region or in different regions. The multiple hosts / virtual machines / containers used to run the code may be distributed in the same available zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0235] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a single region. Communication between two VPCs within the same region, and between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0236] As an example of a hardware functional unit, a data processing device may include at least one computing device, such as a server. Alternatively, the data processing device may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The aforementioned PLD may be implemented using a complex PLD (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0237] The data processing unit comprises multiple computing devices that can be distributed within the same region or in different regions. Similarly, the multiple computing devices can be distributed within the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices can be distributed within the same Virtual Private Cloud (VPC) or multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0238] This application also provides a computing device 130. For example... Figure 14 As shown, the computing device 130 includes a bus 132, a processor 134, a memory 136, and a communication interface 138. The processor 134, the memory 136, and the communication interface 138 communicate with each other via the bus 132. The computing device 130 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 130.

[0239] Bus 132 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 14 The bus 134 is represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 134 may include a path for transmitting information between various components of the computing device 130 (e.g., memory 136, processor 134, communication interface 138).

[0240] The processor 134 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0241] Memory 136 may include volatile memory, such as random access memory (RAM). Processor 134 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0242] The memory 136 stores executable program code, and the processor 134 executes the executable program code to implement the functions of the aforementioned acquisition module and training module, thereby realizing the data processing method applied to the computing device cluster in the above embodiments. That is, the memory 136 stores instructions for executing the data processing method applied to the computing device cluster in the above embodiments.

[0243] The communication interface 138 uses transceiver modules, such as, but not limited to, network interface cards and transceivers, to enable communication between the computing device 130 and other devices or communication networks.

[0244] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0245] like Figure 15 As shown, the computing device cluster includes at least one computing device 130. The memory 136 of one or more computing devices 130 in the computing device cluster may store the same instructions for performing data processing methods.

[0246] In some possible implementations, the memory 136 of one or more computing devices 130 in the computing device cluster may also store partial instructions for executing data processing methods. In other words, a combination of one or more computing devices 130 can jointly execute instructions for executing data processing methods.

[0247] It should be noted that the memory 136 in different computing devices 130 within the computing device cluster can store different instructions, each used to execute a portion of the data processing method's functions. That is, the instructions stored in the memory 136 of different computing devices 130 can implement one or more functions of the acquisition module.

[0248] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN), a local area network (LAN), or similar. Figure 15 One possible implementation is shown. For example... Figure 15 As shown, two computing devices 130A and 130B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this type of possible implementation, the memory 136 in computing device 130A may store instructions for executing the functions of the first processing module. Simultaneously, the memory 136 in computing device 130B may store instructions for executing the functions of the second processing module. Alternatively, the memory 136 in computing device 130A may store instructions for executing some functions of the second processing module, while the memory 136 in computing device 130B may store instructions for executing another part of the functions of the second processing module, and so on.

[0249] It should be understood that Figure 16 The functions of the computing device 130A shown can also be performed by multiple computing devices 130. Similarly, the functions of the computing device 130B can also be performed by multiple computing devices 130.

[0250] This application also provides another computing device cluster. The connection relationships between the computing devices in this computing device cluster can be similarly referred to... Figure 15 The connection method of the computing device cluster. The difference is that the memory 136 of one or more computing devices 130 in the computing device cluster can store the same instructions for executing data processing methods.

[0251] In some possible implementations, the memory 136 of one or more computing devices 130 in the computing device cluster may also store partial instructions for executing data processing methods. In other words, a combination of one or more computing devices 130 can jointly execute instructions for executing data processing methods.

[0252] It should be noted that the memory 136 in different computing devices 130 within the computing device cluster can store different instructions for executing parts of the data processing methods. That is, the instructions stored in the memory 136 of different computing devices 130 can implement one or more functions of the processing module.

[0253] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions capable of running on a computing device or stored on any usable medium. When the computer program product runs on at least one computing device, it causes the at least one computing device to perform a parallel processing method.

[0254] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center that includes one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to perform a parallel processing method.

[0255] This application also provides a chip system including a processor for implementing the steps performed by the aforementioned computing device cluster. In one possible design, the chip system may further include a memory for storing necessary program instructions and data. This chip system may be composed of chips or may include chips and other discrete devices.

[0256] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0257] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0258] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0259] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0260] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A parallel processing method, characterized in that, The method includes: The input data in the target model is segmented to obtain a first segmented data set and a second sliced ​​data set; The first segmented data set and the second sliced ​​data set are processed by a parallel network module to obtain a first branch sequence and a second branch sequence, wherein the first branch sequence and the branch sequence respectively include multiple operators; A set of parallel operators between branches is determined from the first branch sequence and the second branch sequence by dynamic programming. The set of parallel operators between branches includes a first operator from the first branch sequence and a second operator from the second branch sequence. The first operator and the second operator are executed in parallel to obtain the operator execution results.

2. The method according to claim 1, characterized in that, The method further includes: removing commonly accessed operators from the first branch sequence and the second branch sequence to obtain a first branch sequence and a second branch sequence after removing the operators; The step of determining the set of inter-branch parallel operators from the first branch sequence and the second branch sequence through dynamic programming includes: The set of parallel operators between branches is determined by dynamic programming from the first branch sequence after the removal operator and the second branch sequence after the removal operator.

3. The method according to claim 2, characterized in that, The step of removing operators that are accessed in both the first branch sequence and the second branch sequence includes: Starting with the merge operator inserted into the target model, branch label propagation is performed on the first branch sequence and the second branch sequence, and branch label propagation stops at the splitting operator inserted into the target model.

4. The method according to any one of claims 1 to 3, characterized in that, The method further includes: merging consecutive operators of the same type within the first branch sequence and the second branch sequence to obtain a first branch sequence and a second branch sequence after merging operators; The step of determining the set of inter-branch parallel operators from the first branch sequence and the second branch sequence through dynamic programming includes: The set of parallel operators between branches is determined by dynamic programming from the first branch sequence after the merging operator and the second branch sequence after the merging operator.

5. The method according to claim 4, characterized in that, The merging of consecutive operators of the same type within the first branch sequence and the second branch sequence includes: Obtain the runtime of the operators within each sequence of the first branch sequence and the second branch sequence; Obtain the sum of runtime of consecutive operators of the same type within each of the first branch sequence and the second branch sequence; If the total runtime is less than a first runtime threshold, consecutive operators of the same type within the first branch sequence and the second branch sequence are merged.

6. The method according to any one of claims 1 to 5, characterized in that, The step of determining the set of inter-branch parallel operators from the first branch sequence and the second branch sequence through dynamic programming includes: Obtain the first hidden duration for communication and computation between the first (i-1) operators in the first branch sequence and the first j operators in the second branch sequence, where i and j are positive integers; Obtain the second hidden duration for communication and computation between the first i operators in the first branch sequence and the first (j-1) operators in the second branch sequence; Obtain the third hidden duration for communication and computation between the first (i-1) operators in the first branch sequence and the first (j-1) operators in the second branch sequence, and the fourth hidden duration for communication and computation between the i-th operator in the first branch sequence and the j-th operator in the second branch sequence; A fifth hidden duration is obtained based on the first hidden duration, the second hidden duration, the third hidden duration, and the fourth hidden duration. The fifth hidden duration is the hidden duration for communication and calculation between the first i operators in the first branch sequence and the first j operators in the second branch sequence. The maximum duration of the fifth hidden duration is taken as the optimal path of the dynamic programming, and the i-th operator in the first branch sequence is determined as the first operator, and the j-th operator in the second branch sequence is determined as the second operator; Determine that the first operator and the second operator belong to the set of inter-branch parallel operators.

7. The method according to claim 6, characterized in that, When the i-th operator and the j-th operator have the same operator type, the fourth hiding time is 0.

8. The method according to any one of claims 1 to 7, characterized in that, The parallel execution of the first operator and the second operator includes: A first control edge is added between the first operator and the second operator according to the set of inter-branch parallel operators; The first operator and the second operator are scheduled to complete parallel execution according to the first control edge.

9. The method according to any one of claims 1 to 8, characterized in that, The method further includes: The results of the operator execution are merged by merging operators inserted into the target model to obtain a merged result.

10. The method according to any one of claims 1 to 9, characterized in that, The segmentation process of the input data in the target model includes: The input data in the target model is segmented by a segmentation operator inserted into the target model.

11. A parallel processing device, characterized in that, The device includes: The segmentation module is used to segment the input data in the target model to obtain the first segmented data set and the second slice data set; The data processing module is used to process the first segmented data set and the second sliced ​​data set respectively through the parallel network module to obtain a first branch sequence and a second branch sequence, wherein the first branch sequence and the branch sequence respectively include multiple operators; The inter-branch operator merging module is used to determine a set of inter-branch parallel operators from the first branch sequence and the second branch sequence through dynamic programming. The set of inter-branch parallel operators includes: a first operator from the first branch sequence and a second operator from the second branch sequence. The parallel execution module is used to execute the first operator and the second operator in parallel to obtain the operator execution result.

12. A computer-readable storage medium comprising instructions which, when executed on a computer, cause the computer to perform the method of any one of claims 1 to 10.

13. A computer program product comprising instructions that, when run on a computer, cause the computer to perform the method as described in any one of claims 1 to 10.

14. A chip, characterized in that, The device includes one or more interface circuits and one or more processors; the interface circuits are configured to receive signals from the memory of the electronic device and send the signals to the processors, the signals including computer instructions stored in the memory; when the processor executes the computer instructions, the electronic device performs the method of any one of claims 1 to 10.

Images