Data processing method and related device

By replacing the All-Reduce operator with the All-Gather operator in distributed training of AI models, and adjusting the matrix multiplication structure and weight splitting method, the problem of high communication time is solved, the model training efficiency is improved and the memory is satisfied. Constraints enable efficient AI model training.

CN119003957APending Publication Date: 2024-11-22HUAWEI TECH CO LTD
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202310559001.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-17
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

During the distributed training of AI models, communication time consumption accounts for a high proportion, resulting in low model training efficiency. Although the existing quantization and low-rank decomposition methods can compress communication volume, they require additional computing time and are not significantly Improve training efficiency.

Method used

By replacing the all-reduce operator with high complexity, all-reduce operators are collected for the All-Gather operators, and adjusting the splitting method and weight compression of the matrix multiplication structure when necessary, reducing the processing time and memory usage of the communication operator, and improving the Model training efficiency.

Benefits of technology

It effectively reduces communication time, improves the training efficiency of AI models, and ensures that the accuracy of the output value is not affected by adjusting the weight compression ratio, satisfies the video memory constraints, and improves the efficiency of the graphics card usage.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119003957A_ABST
    Figure CN119003957A_ABST
Patent Text Reader

Abstract

The invention discloses a data processing method and a related device, which can be applied to a distributed training scene of an artificial intelligence model. The method comprises the following steps: extracting a first matrix multiplication structure of an artificial intelligence model in a plurality of nodes, wherein the first matrix multiplication structure is a structure formed by multiplying a plurality of matrixes; and determining a first communication operator used by the first matrix multiplication structure, wherein the first communication operator is used for summarizing the first matrix multiplication structures in the plurality of nodes. A first communication operator used by the first matrix multiplication structure is replaced by a second communication operator, and the complexity of the second communication operator is lower than that of the first communication operator. According to the method, the second communication operator with lower complexity is used for replacing the first communication operator, so that the waiting time of the communication operators in the summarizing processing process can be reduced, the communication time consumption can be reduced, and the training efficiency of the model can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of machine learning, and in particular to a data processing method and related devices. Background Art

[0002] With the development of artificial intelligence (AI), large transformer models such as BERT, GPT, and GPT-3 are increasingly used in the pre-training paradigm. The number of parameters of large transformer models can reach billions, tens of billions, or hundreds of billions. During the model training process, a single node is difficult to carry the calculation of such a large number of parameters. Therefore, cross-node distributed training is usually used to distribute the parameters across multiple nodes in a hybrid parallel manner to train the model. In this process, a large number of large matrix multiplication calculations are involved, and large matrix multiplications under hybrid parallelism also require the introduction of a large number of All-Reduce communication operators to summarize the calculation results of each node. The All-Reduce operator has a high complexity, resulting in a high proportion of pure communication time in model training, and the model training efficiency is low.

[0003] At present, reducing communication time consumption is mainly achieved by compressing the communication volume. The main means of compressing communication volume are quantization and low-rank decomposition. Quantization compresses the calculation result of large matrix multiplication into an eight-bit integer (int8), and then restores it to a floating point type after sending it to the target node. Low-rank decomposition decomposes a large matrix into two small matrices, and then restores it to the original matrix after transmitting it to the target node. It can be seen that both quantization and low-rank decomposition require compression and restoration operations, and both compression and restoration operations need to be implemented through calculations, which also introduces additional computational time, resulting in no significant improvement in the training efficiency of the model. Summary of the invention

[0004] The present application provides a data processing method and related devices for reducing the communication time consumed by an AI model during training and improving the training efficiency of the AI ​​model.

[0005] In a first aspect, the present application provides a data processing method, which is applied to a data processing device, and the method includes: the data processing device extracts a first matrix multiplication structure of an artificial intelligence AI model in multiple nodes, and the first matrix multiplication structure is a structure of multiplying multiple matrices; the data processing device determines a first communication operator used by the first matrix multiplication structure, and the first communication operator is used to summarize and process the first matrix multiplication structure in multiple nodes; the data processing device replaces the first communication operator with a second communication operator, and the complexity of the second communication operator is lower than that of the first communication operator.

[0006] In the first aspect of the present application, replacing the first communication operator with a second communication operator with lower complexity can reduce the waiting time of the communication operator during the aggregation processing, thereby reducing communication time and improving the training efficiency of the model.

[0007] In a possible implementation manner of the first aspect, the first communication operator is an All-Reduce operator, and the second communication operator is an All-Gather operator.

[0008] In this possible implementation, specific forms of the first communication operator and the second communication operator are limited, thereby improving the feasibility of the solution.

[0009] In a possible implementation of the first aspect, the first matrix multiplication structure includes a first training sample and a first weight. Under the segmentation requirement of the All-Reduce operator, the segmentation method of the first training sample is row-column segmentation, and the segmentation method of the first weight is row segmentation. In the above steps: before the data processing device replaces the first communication operator with the second communication operator, the method also includes: the data processing device converts the segmentation method of the first training sample into row segmentation; the data processing device converts the segmentation method of the weight into column segmentation.

[0010] In this possible implementation, the segmentation method of the first training sample is converted from segmentation by rows and columns to segmentation by rows only, and the segmentation method of the first weight is converted from segmentation by rows to segmentation by columns, thereby satisfying the segmentation requirement of the All-Gather operator.

[0011] In a possible implementation of the first aspect, the first matrix multiplication structure includes a first training sample and a first weight, and the first weight is divided into row segments under the segmentation requirements of the All-Reduce operator. The above-mentioned step: before the data processing device replaces the first communication operator with the second communication operator, the method also includes: the data processing device compresses the first weight to obtain the second weight; the data processing device converts the segmentation method of the first weight into row segments to obtain a third weight; the data processing device compresses the third weight to obtain a fourth weight; the above-mentioned step: the data processing device replaces the first communication operator with the second communication operator, including: the data processing device converts the first matrix multiplication structure into a second matrix multiplication structure and a third matrix multiplication structure, wherein the second matrix multiplication structure includes the first training sample and the second weight, the third matrix multiplication structure includes the second training sample and the fourth weight, and the second training sample is obtained after the second matrix multiplication structure is aggregated and processed by the All-Reduce operator; the data processing device replaces the All-Reduce operator used in the third matrix multiplication structure with the All-Gather operator.

[0012] Since the All-Reduce operator performs summation and the All-Gather operator performs merging and summarization, the output value processed by the All-Gather operator occupies a large amount of video memory. After the All-Gather operator is used to replace the All-Reduce operator, redundant memory is introduced, which may cause the video memory overflow problem. Therefore, the conversion process of the communication operator is adjusted so that the output value meets the video memory constraint.

[0013] Specifically, the data processing device converts the first matrix multiplication structure into a second matrix multiplication structure and a third matrix multiplication structure. In order to reduce video memory, the second matrix multiplication structure still uses the All-Reduce operator, and only the third matrix multiplication structure is converted into a communication operator. The training sample in the second matrix multiplication structure is still the first training sample. In order to further compress the parameter amount and reduce video memory, the first weight is compressed to obtain the second weight, and the second weight is used as the weight in the second matrix multiplication structure. That is, the second matrix multiplication structure includes the first training sample and the second weight. The splitting method of the second weight is the same as the first weight, which is still row splitting. Therefore, the calculation result of the second matrix multiplication structure is row splitting. The data processing device then summarizes the second matrix multiplication structures of multiple nodes through the All-Reduce operator. Specifically, the calculation results of the second matrix multiplication structures of multiple nodes are added and summarized, and the result obtained is the second training sample, which is the training sample in the third matrix multiplication structure. The splitting method of the second training sample is the same as the calculation result of the second matrix multiplication structure, which is also row splitting.

[0014] The data processing device replaces the communication operator used in the third matrix multiplication structure from the All-Reduce operator to the All-Gather operator. The segmentation requirement of the All-Gather operator is that the training sample is segmented in rows and the weight is segmented in columns. Therefore, the segmentation method of the second training sample does not need to be converted. For the weight, the segmentation method of the first weight is row segmentation, so it is necessary to first convert the segmentation method of the first weight into column segmentation to obtain the third weight. In order to further compress the parameter amount and ensure that the accuracy of the final output value remains unchanged, the third weight is further compressed to obtain the fourth weight, which is the weight in the third matrix multiplication structure. That is, the third matrix multiplication structure includes the second training sample and the fourth weight.

[0015] In this possible implementation, only the communication operator of a part of the structure is replaced with the All-Gather operator, and the amount of parameters is reduced by compressing the weights, so that the memory constraint can be met. In addition, reducing the amount of parameters can reduce the processing time of the communication operator and further reduce the communication time. By adjusting the compression ratio of the two weights, the accuracy of the final output value can be unaffected. In addition, the division method of the training samples and weights in the second matrix multiplication structure does not need to be converted, and the division method of the result (i.e., the second training sample) obtained by the All-Reduce operator summarizing the second matrix multiplication structure also meets the All-Gather operator's division requirements for the training samples, so it is only necessary to convert the division method of the weights in the third matrix multiplication structure, thereby simplifying the conversion process of the division method.

[0016] In a possible implementation manner of the first aspect, the number of rows and columns of each matrix in the multiple matrices is greater than 1000.

[0017] In this possible implementation, the size of the matrix in the first matrix multiplication structure is limited, thereby improving the feasibility of the solution.

[0018] In a possible implementation manner of the first aspect, the AI ​​model is a transformer type model.

[0019] In this possible implementation method, the specific form of the AI ​​model is limited, which improves the feasibility of the solution.

[0020] The second aspect of the present application provides a data processing device, including an extraction unit, a determination unit, and a replacement unit. The extraction unit is used to extract a first matrix multiplication structure of an AI model in multiple nodes, where the first matrix multiplication structure is a structure of multiplying multiple matrices; the determination unit is used to determine a first communication operator used by the first matrix multiplication structure, where the first communication operator is used to aggregate and process the first matrix multiplication structure in multiple nodes; and the replacement unit is used to replace the first communication operator with a second communication operator, where the complexity of the second communication operator is lower than that of the first communication operator.

[0021] In a possible implementation manner of the second aspect, the first communication operator is an All-Reduce operator, and the second communication operator is an All-Gather operator.

[0022] In a possible implementation of the second aspect, the first matrix multiplication structure includes a first training sample and a first weight. Under the segmentation requirements of the All-Reduce operator, the segmentation method of the first training sample is row-column segmentation, and the segmentation method of the first weight is row segmentation. The device also includes: a conversion unit, which is used to convert the segmentation method of the first training sample into row segmentation; the conversion unit is also used to convert the segmentation method of the weight into column segmentation.

[0023] In a possible implementation of the second aspect, the first matrix multiplication structure includes a first training sample and a first weight, and the first weight is divided into row segments under the segmentation requirements of the All-Reduce operator. The device also includes: a compression unit, which is used to compress the first weight to obtain a second weight; the conversion unit is also used to convert the segmentation method of the first weight into row segments to obtain a third weight; the compression unit is also used to compress the third weight to obtain a fourth weight; the replacement unit is specifically used to: convert the first matrix multiplication structure into a second matrix multiplication structure and a third matrix multiplication structure, wherein the second matrix multiplication structure includes the first training sample and the second weight, the third matrix multiplication structure includes the second training sample and the fourth weight, and the second training sample is obtained after the second matrix multiplication structure is aggregated and processed by the All-Reduce operator; and the All-Reduce operator used in the third matrix multiplication structure is replaced with an All-Gather operator.

[0024] In a possible implementation manner of the second aspect, the number of rows and columns of each matrix in the multiple matrices is greater than 1000.

[0025] In a possible implementation of the second aspect, the AI ​​model is a transformer class model.

[0026] The data processing device provided in the second aspect of the present application is used to execute the method described in the first aspect or any possible implementation manner of the first aspect.

[0027] The third aspect of the present application provides a data processing device, including a processor and a memory, wherein the memory is used to store instructions, and the processor is used to obtain the instructions stored in the memory to execute the method described in the first aspect or any possible implementation of the first aspect.

[0028] A fourth aspect of the present application provides a computer-readable storage medium, which includes instructions. When the instructions are executed on a computer, the computer executes the method described in the first aspect or any possible implementation of the first aspect.

[0029] A fifth aspect of the present application provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the method described in the first aspect or any possible implementation of the first aspect.

[0030] In a sixth aspect, the present application provides a chip system, which includes at least one processor and a communication interface, wherein the communication interface and the at least one processor are interconnected via lines, and the at least one processor is used to run computer programs or instructions to execute the method described in the first aspect or any possible implementation of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 A schematic diagram of an application scenario of the data processing method provided in an embodiment of the present application;

[0032] Figure 2 A system structure diagram for the data processing method provided in the embodiment of the present application;

[0033] Figure 3 A schematic diagram of another embodiment of the data processing method provided in the embodiment of the present application;

[0034] Figure 4a Schematic diagram of the processing flow of the All-Reduce operator in the embodiment of the present application;

[0035] Figure 4b Schematic diagram of the processing flow of the All-Gather operator in the embodiment of the present application;

[0036] Figure 4c This is a schematic diagram of the conversion process of the communication operator in the embodiment of the present application;

[0037] Figure 5 A schematic diagram of another embodiment of the data processing method provided in the embodiment of the present application;

[0038] Figure 6 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application;

[0039] Figure 7 Another structural schematic diagram of the data processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0040] The following describes the embodiments of the present application in conjunction with the accompanying drawings. Obviously, the described embodiments are only embodiments of a part of the present application, rather than all embodiments. It is known to those skilled in the art that with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0041] The terms "system" and "network" in the specification and claims of the present application and the above-mentioned drawings can be used interchangeably. Unless otherwise specified, ordinal numbers such as "first" and "second" are used to distinguish multiple objects and are not used to limit the order, timing, priority or importance of multiple objects. It should be understood that the terms used in this way can be interchangeable where appropriate, so that the embodiments described here can be implemented in an order other than the content illustrated or described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0042] To facilitate understanding, the relevant terms and concepts mainly involved in the embodiments of the present application are first introduced below.

[0043] 1. Large Matrix Multiplication

[0044] The product operation between multiple matrices whose row and column numbers are all greater than 1000 is considered to be a large matrix multiplication.

[0045] 2. Hybrid parallelism

[0046] A parallel computing method that divides training samples by nodes and weights by nodes.

[0047] 3. All-Reduce Operator

[0048] A collective communication operation that adds together data of the same shape from multiple nodes into a single aggregated data of the same shape.

[0049] 4. All-Gather operator

[0050] A collective communication operation that merges and concatenates data of the same shape from multiple nodes into a larger shape.

[0051] 5. Distributed training

[0052] A model training method that distributes the number of parameters and the amount of computation across multiple nodes to complete forward propagation and reverse parameter update.

[0053] 6. Transformer

[0054] A deep learning model with multi-head attention structure.

[0055] 7. Pareto Optimal Strategy

[0056] It refers to an ideal state of resource allocation. Assuming that a group of people and allocable resources change from one allocation state to another, at least one person becomes better off without making anyone worse off. This is Pareto improvement or Pareto optimization. The Pareto optimal state means that there is no room for more Pareto improvements. In other words, Pareto improvement is the path and method to achieve Pareto optimality.

[0057] The embodiment of the present application provides a data processing method for reducing the communication time consumption of the AI ​​model during the training process and improving the training efficiency of the AI ​​model. The embodiment of the present application also provides a corresponding device, a computer-readable storage medium, and a computer program product, etc. The following are respectively described.

[0058] See below Figure 1 , Figure 1 A schematic diagram of an application scenario of the data processing method provided in an embodiment of the present application.

[0059] like Figure 1 As shown, the application scenario is an AI model training scenario. The AI ​​model includes a multi-layer forward network and a multi-layer reverse network. Among them, the forward network is used to calculate the training samples to obtain the corresponding output value. The total error, or loss, can be calculated based on the output value and the target value. The total error is used as the input of the reverse network, and the error of each layer of the reverse network is calculated by the back propagation algorithm. The weight update parameters of each layer are calculated based on the error of each layer of the reverse network, and the weights of the corresponding forward network are updated according to the weight update parameters. In other words, the training of the AI ​​model mainly includes three processes, namely the forward propagation process of calculating the output value, the back propagation process of calculating the error, and the process of updating the weight data according to the error.

[0060] With the development of AI, large transformer models have been used more and more widely. The number of parameters (training samples and weights) can reach billions or even tens of billions or hundreds of billions, and a single node can hardly carry the calculation of such a large number of parameters. Therefore, for the training of large models, cross-node distributed training is usually adopted, and hybrid parallelism is used to distribute the number of parameters and the amount of calculation on multiple nodes to complete forward propagation and reverse parameter update. This involves a large number of matrix multiplication calculations, especially large matrix multiplication calculations, which can account for more than 40% of the total calculation amount. In distributed training, the calculation results of large matrix multiplications need to be summarized through communication operators. A large number of large matrix multiplication calculations cause the communication time of the calculation results to account for too high a proportion of the model training time, which can reach 20% to 30%, and the model training efficiency and graphics card utilization efficiency are low.

[0061] In response to the above problems, the embodiment of the present application provides a data processing method, which is executed by a data processing device and can be applied to the distributed training scenario of the AI ​​model. The data processing device uses a communication operator with lower complexity to replace the communication operator with higher complexity used in the matrix multiplication structure in the AI ​​model, thereby reducing the waiting time when the communication operator summarizes the calculation results, thereby reducing the pure communication time consumption and improving the training efficiency of the model.

[0062] See below Figure 2 , Figure 2 A system structure diagram for the data processing method provided in the embodiment of the present application. Figure 2 As shown, the data processing method is mainly implemented by an input perception module, a communication operator conversion module and a computational collaborative optimization module. That is, the data processing device includes an input perception module, a communication operator conversion module and a computational collaborative optimization module. Among them, the input perception module is mainly used to extract the matrix multiplication structure in the AI ​​model according to the configuration file imported by the user, determine the communication operator used in the matrix multiplication structure, and determine the segmentation method of the communication operator. The communication operator conversion module is used to convert the communication operator used in the matrix multiplication structure into a communication operator with lower complexity. The computational collaborative optimization module is used to determine whether the communication operator conversion causes a video memory overflow problem. If so, the matrix multiplication structure is adjusted during the conversion process of the communication operator. After the communication operator conversion is completed, it is determined whether the training efficiency of the AI ​​model is improved. If the training efficiency is improved, the configuration file of the AI ​​model is modified. Specifically, the communication operator used in the matrix multiplication structure in the configuration file is modified, and the optimization process is ended.

[0063] Combine the following Figure 3 The functions of the above three modules are described in detail. Figure 3 , which is a schematic diagram of an embodiment of a data processing method provided in an embodiment of the present application, and the embodiment includes steps 301 to 303.

[0064] 301. Extract a first matrix multiplication structure of the artificial intelligence AI model in multiple nodes, where the first matrix multiplication structure is a structure of multiplying multiple matrices.

[0065] The data processing device reads the configuration file of the AI ​​model through the input perception module. The AI ​​model can be a distributed training model, such as a large transformer model. The configuration file can be a model script, etc. The configuration file includes the model structure and communication strategy, etc. The communication strategy includes the communication operator used and the segmentation requirements of the communication operator.

[0066] The data processing device extracts a first matrix multiplication structure in the AI ​​model according to the configuration file. The first matrix multiplication structure is a structure of multiplying multiple matrices, for example, a structure of multiplying two matrices. The matrix in the first matrix multiplication structure includes a first training sample and a first weight. The first training sample is the input value of the AI ​​model, and the first weight is a coefficient or variable in the model, which is used to control the mapping relationship between the first training sample and the output value.

[0067] Since the training process of the AI ​​model is distributed training, the first matrix multiplication structure (including the first training sample and the first weight) is distributed in multiple nodes in a hybrid parallel manner. In a possible solution, the first matrix multiplication structure in this embodiment is mainly a large matrix multiplication structure, that is, the number of rows and columns of each matrix included in the first matrix multiplication structure is greater than 1000.

[0068] 302. Determine a first communication operator used by a first matrix multiplication structure, where the first communication operator is used to aggregate the first matrix multiplication structures in multiple nodes.

[0069] The data processing device determines the first communication operator used by the first matrix multiplication structure according to the configuration file. Since the first matrix multiplication structure is distributed on multiple nodes, it is necessary to aggregate the first matrix multiplication structures of different nodes in subsequent data processing, specifically to aggregate the calculation results of the first matrix multiplication structures of different nodes (i.e., the product of multiple matrices). The first communication operator plays the role of aggregation processing, and aggregates the first matrix multiplication structures distributed on different nodes. The first communication operator can be a broadcast operator Broadcast, a reduction operator Reduce-Sum, a collection operator Gather, a full reduction operator All-Reduce, and a full collection operator All-Gather, etc.

[0070] 303. Replace the first communication operator with a second communication operator, where the complexity of the second communication operator is lower than that of the first communication operator.

[0071] The data processing device converts the first communication operator used in the first matrix multiplication structure into a second communication operator with lower complexity through the communication operator conversion module. Since the complexity of the second communication operator is low, the waiting time when the first matrix multiplication structure of multiple nodes is aggregated and processed is short, thereby shortening the communication time.

[0072] In a possible solution, the first communication operator is an All-Reduce operator, and the second communication operator is an All-Gather operator.

[0073] The All-Reduce operator aggregates the first matrix multiplication structure and can be combined with Figure 4a To understand. Figure 4aThe first matrix multiplication structure includes a first matrix multiplication structure, an All-Reduce operator, and an output value. The first matrix multiplication structure includes a first training sample, a first weight, and a MatMul function. The MatMul function is a matrix multiplication function, and the first training sample and the first weight are multiplied by the MatMul function. The All-Reduce operator requires the first training sample to be divided into rows and columns, and the first weight to be divided into rows. According to the matrix multiplication rule, the number of rows of the calculation result is equal to the number of rows of the first training sample, and the number of columns is equal to the number of columns of the first weight (the number of columns is 1), that is, the calculation result is divided into rows. The All-Reduce operator adds and summarizes the calculation results of the first matrix multiplication structure of multiple nodes to obtain the output value. The output value is divided in the same way as the calculation result of the first matrix multiplication.

[0074] Figure 4a The parameter quantity of the first training sample in is bs×m, where bs is the dimension of the first training sample and m is the number of columns of the first training sample. The parameter quantity of the first weight is m×n, where m is the number of rows of the first weight and n is the number of columns of the first weight. The communication volume of the All-Reduce operator is bs×n, and the parameter quantity of the output value obtained after the summary processing is also bs×n.

[0075] Compared with the All-Reduce operator that sums up the first matrix multiplication structure, the All-Gather operator merges and concatenates the first matrix multiplication structures of multiple nodes, so no calculation is required and the complexity is lower than that of the All-Reduce operator. The process of the All-Gather operator summarizing the first matrix multiplication structure can be combined with Figure 4b Understand. Figure 4b As shown in the figure, the All-Gather operator requires the first training sample to be split into rows and the first weight to be split into columns. Therefore, the splitting requirements of the All-Gather operator and the All-Reduce operator are inconsistent. Before using the All-Gather operator to replace the All-Reduce operator, the first matrix multiplication structure needs to be adjusted to meet the shape constraint of the All-Gather operator for the input. The All-Gather operator cannot process input data that does not meet the shape constraint.

[0076] Optionally, before replacing the All-Reduce operator, the data processing device converts the segmentation method of the first matrix multiplication structure through the communication operator conversion module. Specifically, the segmentation method of the first training sample is converted from segmentation by rows and columns to segmentation by rows only, and the segmentation method of the first weight is converted from segmentation by rows to segmentation by columns, so as to meet the segmentation requirements of the All-Gather operator.

[0077] After the segmentation requirements of the All-Gather operator are met, the data processing device replaces the All-Reduce operator with the All-Gather operator through the communication operator conversion module. After the communication operator is replaced, the output value will also be modified, and the output value segmentation method is row and column segmentation. Figure 4b Processing flow and Figure 4a Similar, the details will not be repeated here.

[0078] Since the All-Reduce operator performs summation and the All-Gather operator performs merging and summarization, the output value processed by the All-Gather operator occupies a large amount of video memory. After the All-Gather operator is used to replace the All-Reduce operator, redundant memory is introduced, which may cause the video memory to exceed the problem. The data processing device determines whether the communication operator exceeds the video memory after conversion. If so, the conversion process of the communication operator is adjusted through the computing collaborative optimization module so that the output value meets the video memory constraint. Figure 4c Describe the adjustment process in detail.

[0079] like Figure 4c As shown, the data processing device converts the first matrix multiplication structure into a second matrix multiplication structure and a third matrix multiplication structure. In order to reduce video memory, the second matrix multiplication structure still uses the All-Reduce operator, and only the third matrix multiplication structure is converted into a communication operator. The training sample in the second matrix multiplication structure is still the first training sample. In order to further compress the parameter amount and reduce video memory, the first weight is compressed to obtain the second weight, and the second weight is used as the weight in the second matrix multiplication structure. That is, the second matrix multiplication structure includes the first training sample and the second weight. The splitting method of the second weight is the same as the first weight, which is still row splitting. Therefore, the calculation result of the second matrix multiplication structure is row splitting. The data processing device then summarizes the second matrix multiplication structures of multiple nodes through the All-Reduce operator. Specifically, the calculation results of the second matrix multiplication structures of multiple nodes are added and summarized, and the result obtained is the second training sample, which is the training sample in the third matrix multiplication structure. The splitting method of the second training sample is the same as the calculation result of the second matrix multiplication structure, which is also row splitting.

[0080] The data processing device replaces the communication operator used in the third matrix multiplication structure from the All-Reduce operator to the All-Gather operator. The segmentation requirement of the All-Gather operator is that the training sample is segmented in rows and the weight is segmented in columns. Therefore, the segmentation method of the second training sample does not need to be converted. For the weight, the segmentation method of the first weight is row segmentation, so it is necessary to first convert the segmentation method of the first weight into column segmentation to obtain the third weight. In order to further compress the parameter amount and ensure that the accuracy of the final output value remains unchanged, the third weight is further compressed to obtain the fourth weight, which is the weight in the third matrix multiplication structure. That is, the third matrix multiplication structure includes the second training sample and the fourth weight.

[0081] Figure 4c In the example, only the communication operator of some structures is replaced by the All-Gather operator, and the number of parameters is reduced by compressing the weights, so that the video memory constraint can be met. In addition, reducing the number of parameters can reduce the processing time of the communication operator and further reduce the communication time. By adjusting the compression ratio of the two weights, the accuracy of the final output value can be kept unchanged. In addition, the division method of the training samples and weights in the second matrix multiplication structure does not need to be converted, and the division method of the result (i.e., the second training sample) obtained by the All-Reduce operator summarizing the second matrix multiplication structure also meets the division requirements of the All-Gather operator for the training samples. Therefore, it is only necessary to convert the division method of the weights in the third matrix multiplication structure, thereby simplifying the conversion process of the division method.

[0082] Figure 4c In , the parameter amount of the first training sample is bs×m, the parameter amount of the second weight is m×k, bs is the sample dimension of the first training sample, m is the number of columns of the first training sample and the number of rows of the second weight, and k is the number of columns of the second weight. The parameter amount of the fourth weight is k×n, k is the number of rows of the fourth weight, n is the number of columns of the fourth weight, and k needs to be smaller than m and n. The communication volume of the second matrix multiplication structure is bs×k, and the communication volume after the All-Reduce operator is processed is bs×n / mp, where mp is the number of nodes for parallel processing, and bs×n / mp is the parameter amount of the second training sample. The parameter amount of the final output value is bs×n.

[0083] In this embodiment, the second communication operator with lower complexity is used to replace the first communication operator, which can reduce the waiting time in the communication operator aggregation process, thereby reducing the communication time consumption and improving the training efficiency of the model. In addition, when the video memory is exceeded, the parameter amount can be reduced by compressing the weight, and the reduction in the parameter amount can further reduce the communication time consumption, thereby further improving the training efficiency of the model and satisfying the video memory constraint, thereby improving the use efficiency of the graphics card.

[0084] Combine the following Figure 5 The data processing method provided in the embodiments of the present application is briefly described.

[0085] like Figure 5 As shown, the user first imports the model configuration file (such as a model script, etc.) into the data processing device. The input perception module of the data processing device reads the configuration file, and the input perception module includes an operator identification module and a complexity matching module. Specifically, the operator identification module extracts structures involving matrix multiplication, especially structures involving large matrix multiplication, and determines the communication operator used by the matrix multiplication structure. The complexity matching module determines whether the communication operator used in the matrix multiplication structure is a high-complexity operator (such as an All-reduce operator). If so, the high-complexity communication operator is replaced with a low-complexity communication operator (such as an All-Gather operator) through the communication operator conversion module, and the matrix multiplication structure is converted in the segmentation method according to the segmentation requirements of the low-complexity communication operator to meet the segmentation constraints of the low-complexity communication operator.

[0086] If the conversion of the communication operator causes the demand for video memory to increase and thus exceeds the video memory, the weights are adjusted through the computational collaborative optimization module to meet the video memory constraint. Specifically, the shape reorganization module converts the matrix multiplication structure into two small matrix multiplication structures. The first matrix multiplication structure still uses the original communication operator, and the second matrix multiplication structure uses a communication operator with low complexity, thereby reducing video memory. In addition, the weights of the two matrix multiplication structures are compressed, thereby reducing the number of parameters, further reducing the cache, and reducing the processing time of the communication operator, further reducing the communication time. The specific process is described in Figure 4c The details are explained in , and will not be repeated here.

[0087] After the communication operator conversion is completed, the data processing device determines whether the training efficiency of the AI ​​model is improved and the accuracy loss is small after replacing it with a low-complexity communication operator. The specific method can be to make a judgment through the Pareto optimal strategy. If the Pareto optimality is met, the configuration file of the AI ​​model is modified and the AI ​​model is recompiled after modification.

[0088] The above describes the embodiments of the present application from the perspective of the method. The following describes the relevant devices in the embodiments of the present application from the perspective of specific device implementation.

[0089] See also Figure 6 , an embodiment of the present application provides a schematic diagram of a data processing device 600, wherein the data processing device 600 includes an extracting unit 601, a determining unit 602 and a replacing unit 603.

[0090] The extraction unit 601 is used to extract the first matrix multiplication structure of the AI ​​model in multiple nodes, where the first matrix multiplication structure is a structure of multiplying multiple matrices.

[0091] The determination unit 602 is used to determine a first communication operator used by the first matrix multiplication structure, where the first communication operator is used to aggregate the first matrix multiplication structures in multiple nodes.

[0092] The replacement unit 603 is used to replace the first communication operator with a second communication operator, where the complexity of the second communication operator is lower than that of the first communication operator.

[0093] Optionally, the first communication operator is an All-Reduce operator, and the second communication operator is an All-Gather operator.

[0094] Optionally, the first matrix multiplication structure includes training samples and first weights. Under the segmentation requirements of the All-Reduce operator, the segmentation method of the training samples is row-column segmentation, and the segmentation method of the first weights is row segmentation. The device also includes: a conversion unit 604, which is used to convert the segmentation method of the training samples into row segmentation; the conversion unit 604 is also used to convert the segmentation method of the weights into column segmentation.

[0095] Optionally, the first matrix multiplication structure includes a first training sample and a first weight, and the first weight is divided into rows under the division requirement of the All-Reduce operator. The device also includes: a compression unit 605, which is used to compress the first weight to obtain a second weight. The conversion unit 604 is also used to convert the division method of the first weight into row division to obtain a third weight. The compression unit 605 is also used to compress the third weight to obtain a fourth weight. The replacement unit 603 is specifically used to: convert the first matrix multiplication structure into a second matrix multiplication structure and a third matrix multiplication structure, wherein the second matrix multiplication structure includes the first training sample and the second weight, the third matrix multiplication structure includes the second training sample and the fourth weight, and the second training sample is obtained after the second matrix multiplication structure is aggregated and processed by the All-Reduce operator; the All-Reduce operator used in the third matrix multiplication structure is replaced by the All-Gather operator.

[0096] Optionally, the number of rows and columns of each matrix in the multiple matrices is greater than 1000.

[0097] Optionally, the AI ​​model is a transformer class model.

[0098] Each unit in the data processing device 600 performs the above Figure 2 , Figure 3 and Figure 5 The operation of the data processing device in the illustrated embodiment will not be described in detail herein.

[0099] See below Figure 7 , is a possible structural diagram of a data processing device 700 provided in an embodiment of the present application, including a processor 701, a communication interface 702, a memory 703 and a bus 704. The processor 701, the communication interface 702 and the memory 703 are interconnected via the bus 704. In the embodiment of the present application, the processor 701 is used to control and manage the actions of the data processing device. For example, the processor 701 is used to execute Figure 3 The steps performed by the data processing device in the method embodiment are shown. The communication interface 702 is used to support the data processing device to communicate. The memory 703 is used to store the program code and data of the data processing device.

[0100] Among them, the processor 701 can be a central processing unit, a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a transistor logic device, a hardware component or any combination thereof. It can implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the contents disclosed in this application. The processor can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a digital signal processor and a microprocessor, and so on. The bus 704 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0101] The embodiment of the present application also provides a computer-readable storage medium, wherein the computer-readable storage medium includes instructions, and when the instructions are executed on a computer, the computer executes the aforementioned Figure 2 , Figure 3 and Figure 5 The method in the illustrated embodiment.

[0102] The present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the aforementioned Figure 2 , Figure 3 and Figure 5 The method in the illustrated embodiment.

[0103] The embodiment of the present application also provides a chip system, the chip system includes at least one processor and a communication interface, the communication interface and the at least one processor are interconnected through a line, and the at least one processor is used to run a computer program or instruction to execute the aforementioned Figure 2 , Figure 3 and Figure 5 The method in the illustrated embodiment.

[0104] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0105] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0106] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0107] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0108] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), disk or optical disk and other media that can store program code.

Claims

1. A data processing method, It is characterized in that include: Extracting a first matrix multiplication structure of an artificial intelligence AI model in multiple nodes, where the first matrix multiplication structure is a structure of multiplying multiple matrices; Determining a first communication operator used by the first matrix multiplication structure, where the first communication operator is used to aggregate the first matrix multiplication structures in the multiple nodes; The first communication operator is replaced by a second communication operator, where the complexity of the second communication operator is lower than that of the first communication operator.

2. The method according to claim 1, It is characterized in that The first communication operator is an All-Reduce operator, and the second communication operator is an All-Gather operator.

3. The method according to claim 2, It is characterized in that The first matrix multiplication structure includes a first training sample and a first weight. Under the segmentation requirement of the All-Reduce operator, the segmentation method of the first training sample is row-column segmentation, and the segmentation method of the first weight is row segmentation. Before replacing the first communication operator with the second communication operator, the method further includes: Converting the segmentation method of the first training sample into row segmentation; The weight segmentation method is converted into column segmentation.

4. The method according to claim 2, It is characterized in that The first matrix multiplication structure includes a first training sample and a first weight, and under the segmentation requirement of the All-Reduce operator, the segmentation method of the first weight is row segmentation. Before replacing the first communication operator with the second communication operator, the method further includes: Compress the first weight to obtain the second weight; Convert the segmentation method of the first weight into row segmentation to obtain a third weight; compressing the third weight to obtain a fourth weight; The replacing the first communication operator with the second communication operator comprises: Converting the first matrix multiplication structure into a second matrix multiplication structure and a third matrix multiplication structure, wherein the second matrix multiplication structure includes the first training sample and the second weight, the third matrix multiplication structure includes the second training sample and the fourth weight, and the second training sample is obtained after the second matrix multiplication structure is aggregated and processed by the All-Reduce operator; The All-Reduce operator used in the third matrix multiplication structure is replaced by the All-Gather operator.

5. The method according to any one of claims 1 to 4, It is characterized in that The number of rows and columns of each matrix in the multiple matrices is greater than 1000.

6. The method according to any one of claims 1 to 5, It is characterized in that The AI ​​model is a transformer type model.

7. A data processing device, It is characterized in that Includes: An extraction unit, used to extract a first matrix multiplication structure of the AI ​​model in multiple nodes, where the first matrix multiplication structure is a structure of multiplying multiple matrices; a determining unit, configured to determine a first communication operator used by the first matrix multiplication structure, wherein the first communication operator is used to perform aggregation processing on the first matrix multiplication structures in the plurality of nodes; A replacement unit is used to replace the first communication operator with a second communication operator, where the complexity of the second communication operator is lower than that of the first communication operator.

8. The device according to claim 7, It is characterized in that The first communication operator is an All-Reduce operator, and the second communication operator is an All-Gather operator.

9. The device according to claim 8, It is characterized in that The first matrix multiplication structure includes a first training sample and a first weight. Under the segmentation requirement of the All-Reduce operator, the segmentation method of the first training sample is row-column segmentation, and the segmentation method of the first weight is row segmentation. The device also includes: A conversion unit, used for converting the segmentation mode of the first training sample into row segmentation; The conversion unit is further used to convert the weight segmentation method into column segmentation.

10. The device according to claim 8, It is characterized in that The first matrix multiplication structure includes a first training sample and a first weight, and under the segmentation requirement of the All-Reduce operator, the segmentation method of the first weight is row segmentation, and the device further includes: A compression unit, used for compressing the first weight to obtain a second weight; The conversion unit is further used to convert the segmentation mode of the first weight into row segmentation to obtain a third weight; The compression unit is further used to compress the third weight to obtain a fourth weight; The replacement unit is specifically used for: Converting the first matrix multiplication structure into a second matrix multiplication structure and a third matrix multiplication structure, wherein the second matrix multiplication structure includes the first training sample and the second weight, the third matrix multiplication structure includes the second training sample and the fourth weight, and the second training sample is obtained after the second matrix multiplication structure is aggregated and processed by the All-Reduce operator; The All-Reduce operator used in the third matrix multiplication structure is replaced by the All-Gather operator.

11. The device according to any one of claims 7 to 10, It is characterized in that The number of rows and columns of each matrix in the multiple matrices is greater than 1000.

12. The device according to any one of claims 7 to 11, It is characterized in that The AI ​​model is a transformer type model.

13. A data processing device, It is characterized in that include: Processor and memory; The memory is used to store instructions; The processor is configured to execute instructions stored in the memory to implement the method according to any one of claims 1 to 6.

14. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by one or more processors, the method according to any one of claims 1 to 6 is implemented.

15. A computer program product comprising instructions, It is characterized in that When the computer program product is run on a computer, the computer is caused to execute the method according to any one of claims 1 to 6.

Citation Information

Cited By

  • Calculation graph processing method and device and computer readable storage medium

    CN119624748A

  • Processing Method, Device and Computer Readable Storage Medium of Computation Graph

    CN119624748B