Data processing method, device, electronic device and storage medium
By matching operator information and description information on the artificial intelligence chip, generating performance data and sending it to the server, the fine-grained and flexibility problems of performance data acquisition during large-scale operation are solved, and real-time monitoring and optimization of performance is achieved.
Patent Information
- Application Number
- CN202411166287.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-23
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-08-23
AI Technical Summary
During the operation of large models, it is difficult for the existing technology to achieve fine-grained and flexible performance data acquisition to troubleshoot cluster node failures and optimize performance data.
By matching the operator's operator information with the operator description information when executing the specified code segment of the large model on the artificial intelligence chip, and generating performance data based on the successful matching performance indicator value, it supports flexible acquisition timing and indicator configuration. Send to the server for analysis.
It realizes fine-grained collection of performance indicator values of a single operator during large-scale model operation, improves the flexibility and efficiency of acquisition, supports real-time update of operator description information and performance indicators, and helps quickly troubleshoot and optimize performance.
Smart Images

Figure CN118689753B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence technology, and in particular to a data processing method, device, electronic device and storage medium. Background Art
[0002] As the number of parameters of large models jumps from billions to trillions, the cluster size of large models has also expanded dramatically. For large-scale distributed training tasks such as large models, collecting performance data during the operation of large models is the key to troubleshooting node failures in the cluster and optimizing performance. How to flexibly collect performance data during the operation of large models has become a technical problem that needs to be solved urgently. Summary of the invention
[0003] The embodiments of the present application provide a data processing method, device, electronic device and storage medium, which are used to provide a solution for flexibly collecting performance data during the operation of a large model.
[0004] In a first aspect, an embodiment of the present application provides a data processing method, which is applied to an artificial intelligence chip, comprising:
[0005] In the process of executing the specified code segment of the large model, matching the operator information of each operator corresponding to the specified code segment with at least one operator description information;
[0006] After determining that the operator information of the operator successfully matches any operator description information, collecting the performance indicator value of the operator according to the performance indicator corresponding to the operator description information, wherein the operator description information and the performance indicator are configured based on the collection control instruction sent by the control end;
[0007] Based on the performance indicator value of the operator, generate a piece of performance data;
[0008] The performance data is sent to a server.
[0009] In some embodiments, after determining that the operator information of the operator successfully matches any operator description information, the method further includes:
[0010] Matching the calculation timing of the operator with at least one acquisition timing corresponding to the operator description information, wherein the at least one acquisition timing is configured based on the timing control instruction sent by the control terminal;
[0011] If the calculation timing of the operator successfully matches any collection timing, the performance indicator value of the operator is collected according to the performance indicator corresponding to the operator description information.
[0012] In some embodiments, collecting the performance indicator value of the operator according to the performance indicator corresponding to the operator description information includes:
[0013] Selecting, from the performance indicators corresponding to the operator description information, a performance indicator corresponding to a successfully matched acquisition opportunity, wherein the performance indicator corresponding to the acquisition opportunity is configured based on the opportunity control instruction;
[0014] According to the selected performance indicator, the performance indicator value of the operator is collected.
[0015] In some embodiments, the acquisition timing is batch information of a training batch, or the acquisition timing is calculation order information in a training batch.
[0016] In some embodiments, the performance indicator value of the operator includes execution time and also includes:
[0017] Storing the start event and the end event of the operator in a record queue;
[0018] Based on the execution process of the operator, saving the timestamp of the start event and the timestamp of the end event in the record queue;
[0019] The execution duration of the operator is determined according to the timestamp of the start event and the timestamp of the end event.
[0020] In some embodiments, it also includes:
[0021] Determine the start event and the end event with adjacent timestamps in the record queue as a mergeable event pair;
[0022] Merge the mergeable event pairs in the record queue into one event.
[0023] In some embodiments, after determining that the operator information of the operator successfully matches any operator description information, the method further includes:
[0024] Determine the call relationship information corresponding to the operator through the event stack;
[0025] Based on the performance indicator value of the operator, a performance data is generated, including:
[0026] The performance data is generated based on the performance indicator value of the operator and the call relationship information corresponding to the operator.
[0027] In some embodiments, the performance data also includes a device identifier of a device to which the artificial intelligence chip belongs, an operator identifier of the operator, and communication information.
[0028] In some embodiments, sending the performance data to a server includes:
[0029] After the execution of the operator is completed, the performance data stored in the record queue is pushed into the message buffer;
[0030] After the available space of the message buffer is lower than a preset value or the message buffer meets a clearing condition, packing multiple performance data in the message buffer into one message;
[0031] The message is sent to the server through the message middleware.
[0032] In some embodiments, it also includes:
[0033] In response to the acquisition update instruction sent by the control end, the operator description information is updated, and / or the performance indicator corresponding to the operator description information is updated.
[0034] In some embodiments, it also includes:
[0035] In response to the timing update instruction sent by the control end, at least one of the information of the acquisition timing and the performance indicator corresponding to the acquisition timing is updated.
[0036] In a second aspect, an embodiment of the present application provides a data processing method, which is applied to an electronic device, including:
[0037] Acquire multiple pieces of performance data of multiple artificial intelligence chips from the server, wherein each artificial intelligence chip obtains a piece of performance data according to the following steps: in the process of executing a specified code segment of a large model, match the operator information of each operator corresponding to the specified code segment with at least one operator description information, and after determining that the operator information of the operator successfully matches any operator description information, collect the performance indicator value of the operator according to the performance indicator corresponding to the operator description information, and generate a piece of performance data based on the performance indicator value of the operator, wherein the operator description information and the performance indicator are configured based on the collection control instruction sent by the control end;
[0038] Based on the multiple performance data, determine whether there are any abnormalities in the multiple artificial intelligence chips.
[0039] In some embodiments, each piece of performance data further includes a device identifier of a device to which the corresponding artificial intelligence chip belongs, an operator identifier of a corresponding operator, and communication information. Determining whether the plurality of artificial intelligence chips are abnormal based on the plurality of performance data includes:
[0040] Perform communication matching on the operators on the multiple artificial intelligence chips according to the device identifiers, operator identifiers and communication information in the multiple performance data;
[0041] Determine the communication waiting time of each operator pair according to the performance indicator value of each operator pair that is successfully matched;
[0042] Based on the communication waiting time, determine whether there is a communication anomaly in the artificial intelligence chip corresponding to the operator.
[0043] In some embodiments, multiple performance data of each artificial intelligence chip are packaged in at least one message, and the multiple performance data of the artificial intelligence chip are obtained according to the following steps:
[0044] Obtaining from the server the at least one message sent by the artificial intelligence chip;
[0045] The at least one message is unpacked to obtain multiple performance data of the artificial intelligence chip.
[0046] In a third aspect, an embodiment of the present application provides a data processing device, which is applied to an artificial intelligence chip, including:
[0047] A matching module, used for matching the operator information of each operator corresponding to the specified code segment with at least one operator description information during the execution of the specified code segment of the large model;
[0048] A collection module, configured to collect the performance indicator value of the operator according to the performance indicator corresponding to the operator description information after determining that the operator information of the operator successfully matches any operator description information, wherein the operator description information and the performance indicator are configured based on the collection control instruction sent by the control terminal;
[0049] A generating module, used to generate a piece of performance data based on the performance indicator value of the operator;
[0050] The sending module is used to send the performance data to the server.
[0051] In a fourth aspect, an embodiment of the present application provides a data processing device, applied to an electronic device, including:
[0052] An acquisition module is used to acquire multiple performance data of multiple artificial intelligence chips from a server, wherein each artificial intelligence chip obtains a piece of performance data according to the following steps: in the process of executing a specified code segment of a large model, matching the operator information of each operator corresponding to the specified code segment with at least one operator description information, and after determining that the operator information of the operator successfully matches any operator description information, collecting the performance indicator value of the operator according to the performance indicator corresponding to the operator description information, and generating a piece of performance data based on the performance indicator value of the operator, wherein the operator description information and the performance indicator are configured based on the acquisition control instruction sent by the control end;
[0053] A determination module is used to determine whether there are any abnormalities in the multiple artificial intelligence chips based on the multiple performance data.
[0054] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising a memory, an artificial intelligence chip, and a computer program stored in the memory and executable on the artificial intelligence chip, wherein the artificial intelligence chip implements any of the above-mentioned data processing methods when executing the computer program.
[0055] In a sixth aspect, an embodiment of the present application provides an electronic device, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein:
[0056] The memory stores a computer program that can be executed by at least one processor. The computer program is executed by the at least one processor so that the at least one processor can perform any of the above-mentioned data processing methods.
[0057] In a seventh aspect, an embodiment of the present application provides a storage medium. When a computer program in the storage medium is executed by a processor of an electronic device, the electronic device can execute any of the above-mentioned data processing methods.
[0058] In an eighth aspect, an embodiment of the present application provides a computer program product, including a computer program, which implements any of the above-mentioned data processing methods when executed by a processor.
[0059] In an embodiment of the present application, the operator description information and performance indicators of the operator to be collected are configured by the control end. Subsequently, when the artificial intelligence chip executes the specified code segment of the large model, the operator information of each operator corresponding to the specified code segment can be matched with each operator description information. If the operator information of the operator successfully matches any operator description information, the performance indicator value of the operator can be collected according to the performance indicator corresponding to the operator description information, and a performance data can be generated based on the performance indicator value of the operator, and the performance data can be sent to the server. In this way, the performance indicator value of a single operator can be collected during the operation of the large model, and the collection granularity of the performance data is relatively fine. In addition, the configured operator description information and performance indicators are relatively easy to change, and the collection flexibility is also relatively good. BRIEF DESCRIPTION OF THE DRAWINGS
[0060] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0061] Figure 1 A schematic diagram of the structure of an artificial intelligence chip provided in an embodiment of the present application;
[0062] Figure 2 A flowchart of a data processing method provided in an embodiment of the present application;
[0063] Figure 3 A schematic diagram of recording events in a queue provided in an embodiment of the present application;
[0064] Figure 4 A flowchart of another data processing method provided in an embodiment of the present application;
[0065] Figure 5 A schematic diagram of a performance data collection process provided in an embodiment of the present application;
[0066] Figure 6 An internal schematic diagram of an EVENT STACK provided in an embodiment of the present application;
[0067] Figure 7 Another data processing method provided in the embodiment of the present application;
[0068] Figure 8 A method flow for determining whether there is a communication anomaly provided in an embodiment of the present application;
[0069] Fig. 9 A schematic diagram of a process for determining the communication waiting time of an operator provided in an embodiment of the present application;
[0070] Fig.10 A schematic diagram of a performance data analysis system provided in an embodiment of the present application;
[0071] Fig.11 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application;
[0072] Fig.12 A schematic diagram of the structure of another data processing device provided in an embodiment of the present application;
[0073] Fig.13 A schematic diagram of the hardware structure of an electronic device for implementing a data processing method provided in an embodiment of the present application. DETAILED DESCRIPTION
[0074] In order to provide a solution for flexibly collecting performance data during the operation of a large model, an embodiment of the present application provides a data processing method, device, electronic device and storage medium.
[0075] The preferred embodiments of the present application are described below in conjunction with the drawings in the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application. In addition, the embodiments and features in the embodiments of the present application may be combined with each other if there is no conflict.
[0076] See also Figure 1 , Figure 1 A structural diagram of an artificial intelligence chip 100 provided in an embodiment of the present application at least includes: a video memory 101, an on-chip cache 102 and multiple computing units 103.
[0077] The video memory 101 may be a high bandwidth memory (HBM) or other types of memory. The on-chip cache 102 is a temporary memory with a smaller capacity than the video memory 101 but a faster data read and write speed than the video memory 101 .
[0078] The computing unit 103 may be used to execute various types of operators, including but not limited to: a forward propagation operator, a backward propagation operator, and a communication operator.
[0079] The computing unit 103 can read and write data to the video memory 101, and can also read and write data to the on-chip cache 102. For example, the computing unit 103 can read data from the video memory 101 for calculation, or read data from the video memory 101, then write the read data to the on-chip cache 102, and then read data from the on-chip cache 102 for calculation.
[0080] In addition to the above-mentioned structure, the artificial intelligence chip 100 in the embodiment of the present application may also include other structures, which are not specifically limited in the embodiment of the present application.
[0081] In addition, the artificial intelligence chip 100 can be: a graphics processing unit (GPU), a general-purpose graphics processing unit (GPGPU), a neural network processor (NNPU), a domain specific architecture (DSA), etc.
[0082] The embodiments of this application are based on Figure 1 The artificial intelligence chip 100 shown provides a data processing method, such as Figure 2 As shown, the process of the method is executed by the artificial intelligence chip 100, and includes the following steps:
[0083] In step 201, during the execution of a specified code segment of a large model, operator information of each operator corresponding to the specified code segment is matched with at least one operator description information.
[0084] Among them, the large model refers to a model trained through large-scale distributed training tasks. The large-scale distributed training tasks are usually deployed in a cluster. Each node in the cluster can be an artificial intelligence chip 100, and the large model can be a large language model, a large image model, etc.
[0085] Generally, the designated code segment is a relatively important code segment in the large model, which can be determined by the technician according to the actual business of the large model, and the code location of the code segment can be indicated by the plug-in method. The operator information of the operator includes the operator name, operator type, etc., among which the operator type includes forward operator, backward operator, communication operator, etc. Each operator description information is used to describe the operator information of the operator to be collected (that is, to describe what kind of operator needs to be collected), such as the name, type, and calculation characteristics of the operator.
[0086] In step 202, if the operator information of the operator successfully matches any operator description information, the performance indicator value of the operator is collected according to the performance indicator corresponding to the operator description information, wherein the operator description information and the performance indicator are configured based on the collection control instruction sent by the control end.
[0087] Among them, collecting the performance indicator value of the operator refers to recording the indicator values of the operator under various performance indicators, and the collection control instruction is used to control the operator information of the operator to be collected (that is, control what kind of operator to collect) and the performance indicator of the operator to be collected (that is, control which performance indicators to collect).
[0088] The following is an example of how to collect performance data.
[0089] Assume that during the training of a large model, data collection is required for performance indicators 1, 2, 3, and 4 of the forward operator. Then, the operator description information can be pre-set through the control end: forward class, and the operator description information: performance indicators corresponding to the forward class: performance indicator 1, performance indicator 2, performance indicator 3, and performance indicator 4.
[0090] Subsequently, when a forward operator op1 corresponding to the specified code segment is executed, the operator information of op1 will be successfully matched with the operator description information: forward class, and the performance indicator values of performance indicator 1, performance indicator 2, performance indicator 3, and performance indicator 4 corresponding to the forward class can be collected for op1. In this way, performance data will be collected once each time op1 is executed, and the performance indicators collected each time are the same.
[0091] In some embodiments, the performance indicator value of the operator may include the execution start time and the execution duration, such as the GPU execution duration.
[0092] In specific implementation, the start event and the end event of the operator can be stored in a record queue, and based on the execution process of the operator, the timestamp of the start event and the timestamp of the end event can be saved in the record queue, and then the execution duration of the operator can be determined according to the timestamp of the start event and the timestamp of the end event.
[0093] See also Figure 3 , Figure 3 A schematic diagram of recording events in a queue provided in an embodiment of the present application, where event E1 and event E2 are the start event and end event of operator 1, respectively, and event E3 and event E4 are the start event and end event of operator 2, respectively.
[0094] By comparing the difference between the timestamps of the events, it is found that event E2 and event E3 are adjacent, that is, the difference between the timestamp of event E2 and the timestamp of event E3 is 1. Therefore, event E2 and event E3 are determined to be a mergeable event pair, and then event E2 and event E3 can be merged into one event E2, and the event E4 in the record queue is numbered E3.
[0095] Subsequently, the execution time of operator 1 = the timestamp of event E2 - the timestamp of event E1, and the execution time of operator 2 = the timestamp of event E3 - the timestamp of event E2.
[0096] Since measuring time consumption (i.e., execution time) on an AI chip has overhead, and the overhead is positively correlated with the number of events, the above method can effectively reduce the number of events, thereby reducing the overhead required for measuring time consumption.
[0097] In some embodiments, if a collection update instruction is received from the control end, the operator description information can be updated based on the collection update instruction, and / or the performance indicators corresponding to the operator description information can be updated. That is, during the execution of a large model, such as training or reasoning, the technician can update at least one of the operator description information and performance indicators of the operator to be collected through the control end according to actual needs. In this way, the operator description information and performance indicators of the operator to be collected can be updated in real time during the execution of the large model, which can further improve the flexibility of performance data collection.
[0098] In step 203, a piece of performance data is generated based on the performance indicator value of the operator.
[0099] In some embodiments, the performance indicator values of the operators may be concatenated in a certain order as a piece of performance data.
[0100] In some embodiments, after determining that the operator successfully matches the operator information of any operator description information, the call relationship information corresponding to the operator can also be determined through the event stack, and then a performance data is generated based on the performance indicator value and the call relationship information of the operator. For example, the performance indicator value and the call relationship information of the operator are spliced in a certain order as a performance data.
[0101] In some embodiments, a performance data in any of the above methods may also include the device identifier of the device to which the artificial intelligence chip belongs, the operator identifier of the operator, and communication information, wherein the operator identifier is such as the operator name op_name, and the communication information is such as the communication group type ProcessGroup, the global count value count (used to identify which operators communicate at the same counting moment), etc.
[0102] In step 204, the performance data is sent to the server.
[0103] In actual applications, the performance index value of the operator is calculated as the operator is executed. Before the operator is executed, the relevant calculation information of the performance index can be stored in the record queue. When the operator is executed, the calculation of the performance index value is also completed, and then a complete performance data is obtained. Considering that the number of a performance data is relatively small, the performance data stored in the record queue can be pushed into the message buffer for cache processing. Later, when the available space in the message buffer is lower than the preset value or the message buffer meets the clearing condition, the multiple performance data in the message buffer are packaged into a message, and then the message is sent to the server with low latency through the message middleware such as ZeroMQ.
[0104] In this way, performance data with relatively small data volume are first cached one by one in the message buffer, and then multiple performance data are packaged and sent to the server with low latency through the message middleware. This can not only improve data transmission efficiency but also reduce data transmission delay.
[0105] In the above embodiment, operator description information and performance indicators are configured for each type or category of operators to be collected. Considering that in actual applications, a large model will undergo multiple rounds of training, an operator will calculate samples of multiple training batches, and the operator will perform multiple calculations in one training batch. For an operator, sometimes it may be necessary to collect performance data only for part of the training batches, or only for some calculations in part of the training batches. For this purpose, a collection timing can also be set for the operator to control the performance data collection only at a specific calculation timing of the operator.
[0106] For this purpose, see Figure 4 , Figure 4A flowchart of another data processing method provided in an embodiment of the present application, which is executed by the artificial intelligence chip 100, includes the following steps:
[0107] In step 401, during the execution of a specified code segment of a large model, operator information of each operator corresponding to the specified code segment is matched with at least one operator description information.
[0108] In step 402, after determining that the operator information of the operator successfully matches any operator description information, the calculation timing of the operator is matched with at least one acquisition timing corresponding to the operator description information, wherein the operator description information and the performance indicator are configured based on the acquisition control instruction sent by the control end, and at least one acquisition timing is configured based on the timing control instruction sent by the control end.
[0109] Among them, the calculation timing of the operator includes the samples of which training batch the operator is currently calculating and the calculation order in the corresponding training batch. Each collection timing is used to describe a performance data collection timing, such as performance data collection in a specified training batch, performance data collection at a specific calculation of a specified training batch, etc. The timing control instruction is used to control the timing of performance data collection for the operator.
[0110] In step 403, if the calculation timing of the operator successfully matches any collection timing, the performance indicator value of the operator is collected according to the performance indicator corresponding to the operator description information.
[0111] In some embodiments, whenever the operator information of an operator successfully matches any operator description information, and the calculation timing of the operator successfully matches a collection timing corresponding to the operator description information, the performance indicator value of the operator is collected according to the performance indicator corresponding to the operator description information.
[0112] For example, during the training of a large model, data collection is required for performance indicators 1, 2, 3, and 4 of the backward operator, but data collection is only required in odd training batches, and no data collection is required in even training batches. Then, the operator description information can be pre-set through the control end: backward class, and the collection time corresponding to the backward class can be set: odd training batch, and performance indicators 1, 2, 3, and 4 can be set for the backward class.
[0113] Subsequently, when a backward operator op2 corresponding to the specified code segment is executed, op2 will successfully match the operator description information: backward class. If the training batch currently processed by op2 is odd, the performance indicator values of op2 under performance indicator 1, performance indicator 2, performance indicator 3, and performance indicator 4 can be collected. If the training batch currently processed by op2 is even, performance data collection is not performed.
[0114] In this way, for operators that match the operator description information, data collection can be performed only under specified training batches, and the collection flexibility is better.
[0115] In some embodiments, when the operator information of an operator successfully matches any operator description information, and the calculation timing of the operator successfully matches a collection timing corresponding to the operator description information, the performance indicator corresponding to the successfully matched collection timing can also be selected from the performance indicators corresponding to the operator description information, and then the performance indicator value of the operator is collected according to the selected performance indicator, wherein the performance indicator corresponding to the collection timing is configured based on the timing control instruction, and the collection timing can be the batch information of the training batch, and can also be the calculation order information in the training batch.
[0116] Taking the batch information of the training batch as an example, assuming that during the large model training process, for the backward operator, it is necessary to collect data for performance indicators 1 and performance indicators 2 in odd-numbered training batches, and to collect data for performance indicators 3 and performance indicators 4 in even-numbered training batches.
[0117] Then, you can set the operator description information through the control end: backward class, set the collection time corresponding to the backward class {odd training batch, even training batch}, and set performance indicators 1 and 2 for odd training batches, and set performance indicators 3 and 4 for even training batches.
[0118] Subsequently, when a backward operator op3 corresponding to the specified code segment is executed, op3 will successfully match the operator description information: backward class. If the training batch currently processed by op3 is odd, the performance indicator values of op3 under performance indicator 1 and performance indicator 2 can be collected. If the training batch currently processed by op3 is even, the performance indicator values of op3 under performance indicator 3 and performance indicator 4 can be collected.
[0119] Taking the case where the collection timing is the calculation order information in the training batch as an example, assuming that during the large model training process, for the backward operator, data collection is required for performance indicator 1, performance indicator 2, performance indicator 3, and performance indicator 4 in the odd-numbered calculations of the odd-numbered training batches, and no data collection is performed in the even-numbered training batches.
[0120] Then, you can set the operator description information through the control end: backward class, set the collection time corresponding to the backward class {single calculation of a single training batch}, and set performance index 1, performance index 2, performance index 3, and performance index 4 for the single calculation of a single training batch.
[0121] Subsequently, when a backward operator op4 corresponding to the specified code segment is executed, op4 will successfully match the operator description information: backward class. If op4 currently processes an odd number of calculations of an odd training batch, the performance indicator values of op4 under performance indicator 1, performance indicator 2, performance indicator 3, and performance indicator 4 can be collected. If op4 currently processes an even number of calculations of an odd training batch, or op4 currently processes an even number of training batches, then data collection is not required.
[0122] In this way, for operators that match the operator description information, data collection can be performed in some training batches and not in others. Different performance indicators can be collected for different training batches, and it can be selected which training batches and which calculations to collect data for. The collection flexibility is relatively high.
[0123] In some embodiments, if a timing update instruction is received from the control end, the collection timing can be updated based on the timing update instruction, and / or the performance indicator corresponding to the collection timing can be updated. In this way, it supports real-time updating of at least one of the collection timing of the operator to be collected and the performance indicator corresponding to the collection timing during the operation of the large model, which can further improve the flexibility of performance data collection.
[0124] In step 404, a piece of performance data is generated based on the performance indicator value of the operator.
[0125] In step 405, the performance data is sent to the server.
[0126] In the embodiment of the present application, for each operator to be collected, not only the operator description information and the performance indicator corresponding to the operator description information can be set, but also the collection time of the performance data can be set, and the performance indicators used at different collection times can also be different, so the collection flexibility is relatively high.
[0127] See also Figure 5 , Figure 5 A schematic diagram of a performance data collection process provided in an embodiment of the present application mainly includes three parts: a performance data collection module Perf Data Collector, a communication module Communicator and a controller Controller. These three parts are introduced below respectively.
[0128] 1. Perf Data Collector, including the Record API, the event stack EVENTSTACK, and the Record Queue. In actual applications, the training code of a large model can be pre-set with an insertion point. One insertion point corresponds to one code segment. When running to a code segment corresponding to an insertion point (that is, a specified code segment):
[0129] Record API is used to match the operator information of each operator corresponding to the code segment with the description information of each operator. If the operator information of the operator successfully matches any operator description information, it is determined that the operator meets the collection conditions, that is, it is the operator that needs to perform data collection. After that, the calculation timing of the operator can also be matched with each collection timing corresponding to the operator description information. If the calculation timing of the operator successfully matches any collection timing, it is determined that data collection is currently required for the operator, and then the performance indicator value of the operator can be collected according to the performance indicator corresponding to the operator description information.
[0130] EVENT STACK, used to save the stack information of the operator, including the operator's events Event: such as the start event Event and the end event Event. Figure 6 , Figure 6 This is an internal schematic diagram of an EVENT STACK provided in an embodiment of the present application, which sequentially stores the start of function 1 FUNC_1_START, the start of function 2 FUNC_2_START, the start of function 3 FUNC_3_START, the end of function 3 FUNC_3_END, the end of function 2 FUNC_2_END, and the end of function 1 FUNC_1_END. Function is a concept on the central processing unit (CPU), and operator is a concept on the GPU. Operators and functions are one-to-one corresponding. Here, it is assumed that function 1 corresponds to operator 1, function 2 corresponds to operator 2, and function 3 corresponds to operator 3. Then according to Figure 6 It can be seen that the calling relationship between the three operators is: operator 1 calls operator 2, and operator 2 calls operator 3. Such a stack STACK structure is used to support nested performance measurement.
[0131] Record Queue is used to save the events corresponding to the operator (including at least the start event Even and the end event Even), and save the timestamp of each event Even. It is also used to save the value of each performance indicator of the operator, as well as the device identification, operator identification and communication information.
[0132] 2. Communicator, including message buffer MessageBuffer, packing area Pack and message middleware ZeroMQ. Since the number of performance data in a piece is relatively small, the efficiency of transmitting performance data one by one is relatively low. In order to optimize the transmission performance, the performance data can be pushed to MessageBuffer first. After MessageBuffer is full (that is, the available space is lower than the preset value) or flushed (the clearing condition is met), each performance data in MessageBuffer is packed, that is, each performance data in MessageBuffer is packaged and spliced into a whole message Message, and a message header is added, such as the device ID of the device to which the artificial intelligence chip belongs, the operator ID of the operator, communication information, etc., and then the Message is sent to the kafka process (kafka process) with low latency through ZeroMQ, and the kafka process sends the Message to the kafka server (kafka server). Among them, the kafka process is an independently running process and will not affect the efficiency of the training process.
[0133] 3. Controller: It includes the record controller and the schedule controller. The record controller is used to control the collection strategy, that is, to control what kind of operators to collect, and which performance indicators of the operators to collect, such as GPU time consumption, CPU time consumption, GPU memory status, CPU memory status, etc., to support richer performance data collection. The schedule controller is used to control the collection timing, that is, which batches to collect, how many times to collect, and the rules for continuous collection. The Record API can collect the performance indicator values of operators under the joint control of the record controller and the schedule controller. Generally, the control command cmd of the record controller and the schedule controller comes from the control end, such as the kafka server or other command terminal. The cmd of the control end first reaches the kafkaprocess, and the kafka process sends the cmd to the record controller or schedulecontroller through ZeroMQ. After receiving the cmd, the record controller or schedule controller will feedback the response information ask to the control end through ZeroMQ to ensure that the command is successfully sent.
[0134] In the embodiment of the present application, the performance collection data is buffered by MessageBuffer and ZeroMQ at two levels, and the batch processing of the kafka process reaches the kafka server, which can not only improve the data transmission efficiency, but also reduce the data transmission delay, and the impact on the performance is no more than one thousandth, which is an extremely low-overhead performance collection solution. In addition, the performance data collection can be independently controlled from two orthogonal dimensions (the record controller controls the collection from the dimension of collection conditions, and the schedule controller controls the collection from the dimension of collection timing), which is a flexible real-time performance data collection control solution, which is very suitable for large-scale machine learning training scenarios such as large models, and can well balance the low overhead of long-term monitoring and short-term performance analysis needs.
[0135] Accordingly, the embodiment of the present application also provides a data processing method for processing the performance data collected in the above process. Figure 7 , Figure 7 A flowchart of another data processing method provided in an embodiment of the present application, which is applied to an electronic device and includes the following steps.
[0136] In step 701, multiple performance data of multiple artificial intelligence chips are obtained from the server, wherein each artificial intelligence chip obtains a piece of performance data according to the following steps: in the process of executing a specified code segment of a large model, the operator information of each operator corresponding to the specified code segment is matched with at least one operator description information; after determining that the operator information of the operator successfully matches any operator description information, the performance indicator value of the operator is collected according to the performance indicator corresponding to the operator description information; based on the performance indicator value of the operator, a piece of performance data is generated, and the operator description information and the performance indicator are configured based on the collection control instruction sent by the control end.
[0137] Among them, multiple artificial intelligence chips constitute a cluster that executes large-model distributed tasks.
[0138] In some embodiments, multiple performance data of each artificial intelligence chip are packaged in at least one message, and the multiple performance data of the artificial intelligence chip can be obtained according to the following steps: obtain at least one message sent by the artificial intelligence chip from the server, unpack the at least one message, and obtain multiple performance data of the artificial intelligence chip.
[0139] In step 702, it is determined whether there are any abnormalities in the multiple artificial intelligence chips based on the multiple performance data.
[0140] In actual applications, each piece of performance data may also include the device ID of the device to which the corresponding artificial intelligence chip belongs, the operator ID of the corresponding operator, and communication information. Figure 8, can be based on Figure 8 The process shown in the figure determines whether there are communication anomalies among multiple artificial intelligence chips, including the following steps.
[0141] In step 7021, communication matching is performed on operators on multiple artificial intelligence chips according to device identifiers, operator identifiers, and communication information in multiple performance data.
[0142] Among them, the operator identifier includes the operator name, and the communication information includes the communication group type and count (global count value of the operator).
[0143] Generally, when operators on multiple AI chips are matched for communication, some matching conditions are set, such as two operators are located on different AI chips, the communication group type of the two operators is the same, the count is the same, and the operator names of the two operators match. Based on these matching conditions, operator pairs that communicate on multiple AI chips can be found.
[0144] In step 7022, the communication waiting time of the operator pair is determined according to the performance indicator value of each successfully matched operator pair.
[0145] Generally, the performance data of each operator includes the start timestamp and execution duration of the operator. For each successfully matched operator pair, the communication waiting time of the operator pair can be calculated based on the start timestamp and execution duration of each operator pair.
[0146] In step 7023, based on the communication waiting time, it is determined whether there is a communication anomaly in the artificial intelligence chip corresponding to the operator.
[0147] For example, if the communication waiting time exceeds the set time, it can be determined that there is a communication anomaly in the artificial intelligence chip corresponding to the sender operator in the operator pair.
[0148] The above process is illustrated below with an example.
[0149] In actual applications, each performance data after unpacking will include: device ID, operator name, operator start timestamp, operator execution time, operator communication group type, ProcessGroup, and operator global count value (operators communicating at the same time have the same global count value).
[0150] See also Fig. 9 , Fig. 9 A schematic diagram of a process for determining the communication waiting time of an operator provided in an embodiment of the present application, there are 4 devices in total, the device identifiers are: Device0~Device3, where:
[0151] The performance data on Device0 is:
[0152] op_name: send_grad;
[0153] time_ts: xxxx;
[0154] time_dur: xxxx;
[0155] ProcessGroup:pp;
[0156] count: 56;
[0157] The performance data on Device1 is:
[0158] op_name: recv_grad;
[0159] time_ts: xxxx;
[0160] time_dur: xxxx;
[0161] ProcessGroup:pp;
[0162] Count: 56.
[0163] The performance data on Device2 is:
[0164] op_name: recv_active;
[0165] time_ts: xxxx;
[0166] time_dur: xxxx;
[0167] ProcessGroup:pp;
[0168] Count: 77.
[0169] The performance data on Device 3 is:
[0170] op_name: send_active;
[0171] time_ts: xxxx;
[0172] time_dur: xxxx;
[0173] ProcessGroup:pp;
[0174] Count: 77.
[0175] First, operators are matched for communication, that is, operators that perform communication together are matched together. Matching conditions include ProcessGroup equality, count equality, and op_name matching. The op_name matching rule is related to the operator type. For example, send_grad and recv_grad are one-to-one matching, and recv_active and send_active are one-to-one matching. Fig. 9 After matching the operators in , send_grad on Device0 and recv_grad on Device1 match successfully, and recv_active on Device2 and send_active on Device3 match successfully.
[0176] Then, for each operator pair that matches successfully, determine the communication waiting time of the operator pair. Take send_grad on Device0 and recv_grad on Device1 as an example. The minimum value of time_dur of send_grad on Device0 and time_dur of recv_grad on Device1 is taken as the actual communication time time_comm. Subtract time_comm from time_dur of send_grad on Device0 to get its own communication waiting time. Subtract time_comm from time_dur of recv_grad on Device1 to get its own communication waiting time. The communication waiting time of the two operators on Device2 and Device3 can also be calculated in this way, which will not be repeated here.
[0177] See also Fig.10 , Fig.10 A schematic diagram of a performance data analysis system provided in an embodiment of the present application,
[0178] The entire system is centered around the Kafka server, and the large model cluster will continuously send messages to the Kafka server.
[0179] The preprocessing module PreProcess is used to obtain messages from the Kafka server, unpack the messages into multiple performance data, identify the waiting time of the communication operator based on these multiple performance data, and return these multiple performance data to the Kafka server.
[0180] The visualization processing module VisualizationProcess is used to read multiple performance data after PreProcesse unpacking from the Kafka server, and use the Timeline Generator to convert multiple performance data into the Chrome tracing format. After that, the timeline view can be opened directly in the browser.
[0181] The monitoring service module Monitor server is used to read multiple performance data after PreProcesse unpacking from the Kafka server in real time, convert multiple performance data into required monitoring indicators, and can receive viewing requests from the monitoring client monitorclient, send monitoring indicators to the monitor client, and then display them through visualization software.
[0182] The control client can be used to send control commands cmd to the kafka server. The cmd will be automatically processed by the kafka process on the corresponding device. You can also send query commands to the kafka process on the corresponding device to query the running status of the cmd.
[0183] The embodiment of the present application provides a real-time performance data analysis solution that can identify the communication waiting time of the operator in real time, helping users to more accurately judge the communication performance of the artificial intelligence chip during the large model training process.
[0184] Based on the same technical concept, the embodiment of the present application also provides two data processing devices. The principles of solving the problems by the two data processing devices are similar to those of the above-mentioned two data processing methods. Therefore, the implementation of the data processing devices can refer to the implementation of the data processing methods, and the repeated parts will not be repeated.
[0185] See also Fig.11 , Fig.11 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application includes:
[0186] A matching module 1101 is used to match the operator information of each operator corresponding to the specified code segment with at least one operator description information during the execution of the specified code segment of the large model;
[0187] A collection module 1102 is used to collect the performance indicator value of the operator according to the performance indicator corresponding to the operator description information after determining that the operator information of the operator successfully matches any operator description information, wherein the operator description information and the performance indicator are configured based on the collection control instruction sent by the control terminal;
[0188] A generating module 1103, configured to generate a piece of performance data based on the performance indicator value of the operator;
[0189] The sending module 1104 is used to send the performance data to the server.
[0190] In some embodiments, the matching module 1101 is further configured to match the calculation timing of the operator with at least one acquisition timing corresponding to the operator description information after determining that the operator information of the operator successfully matches any operator description information, wherein the at least one acquisition timing is configured based on the timing control instruction sent by the control terminal;
[0191] The collection module 1102 is further configured to collect the performance indicator value of the operator according to the performance indicator corresponding to the operator description information if the calculation timing of the operator successfully matches any collection timing.
[0192] In some embodiments, the acquisition module 1102 is specifically used to:
[0193] Selecting, from the performance indicators corresponding to the operator description information, a performance indicator corresponding to a successfully matched acquisition opportunity, wherein the performance indicator corresponding to the acquisition opportunity is configured based on the opportunity control instruction;
[0194] According to the selected performance indicator, the performance indicator value of the operator is collected.
[0195] In some embodiments, the acquisition timing is batch information of a training batch, or the acquisition timing is calculation order information in a training batch.
[0196] In some embodiments, the performance indicator value of the operator includes execution time, and the acquisition module 1102 is further used to:
[0197] Storing the start event and the end event of the operator in a record queue;
[0198] Based on the execution process of the operator, saving the timestamp of the start event and the timestamp of the end event in the record queue;
[0199] The execution duration of the operator is determined according to the timestamp of the start event and the timestamp of the end event.
[0200] In some embodiments, an optimization module 1105 is further included for:
[0201] Determine the start event and the end event with adjacent timestamps in the record queue as a mergeable event pair;
[0202] Merge the mergeable event pairs in the record queue into one event.
[0203] In some embodiments, the acquisition module 1102 is further configured to determine the call relationship information corresponding to the operator through the event stack after determining that the operator information of the operator successfully matches any operator description information;
[0204] The generating module 1103 is further configured to generate the performance data based on the performance indicator value of the operator and the call relationship information corresponding to the operator.
[0205] In some embodiments, the performance data also includes a device identifier of a device to which the artificial intelligence chip belongs, an operator identifier of the operator, and communication information.
[0206] In some embodiments, the sending module 1104 is specifically used to:
[0207] After the execution of the operator is completed, the performance data stored in the record queue is pushed into the message buffer;
[0208] After the available space of the message buffer is lower than a preset value or the message buffer meets a clearing condition, packing multiple performance data in the message buffer into one message;
[0209] The message is sent to the server through the message middleware.
[0210] In some embodiments, an update module 1106 is further included, which is used to:
[0211] In response to the acquisition update instruction sent by the control end, at least one of the operator description information and the performance indicator corresponding to the operator description information is updated.
[0212] In some embodiments, an update module 1106 is further included, which is used to:
[0213] In response to the timing update instruction sent by the control end, at least one of the information of the acquisition timing and the performance indicator corresponding to the acquisition timing is updated.
[0214] See also Fig.12 , Fig.12 A structural diagram of another data processing device provided in an embodiment of the present application includes:
[0215] The acquisition module 1201 is used to acquire multiple performance data of multiple artificial intelligence chips from the server, wherein each artificial intelligence chip obtains a piece of performance data according to the following steps: in the process of executing a specified code segment of a large model, matching the operator information of each operator corresponding to the specified code segment with at least one operator description information, and after determining that the operator information of the operator successfully matches any operator description information, collecting the performance indicator value of the operator according to the performance indicator corresponding to the operator description information, and generating a piece of performance data based on the performance indicator value of the operator, wherein the operator description information and the performance indicator are configured based on the acquisition control instruction sent by the control end;
[0216] The determination module 1202 is used to determine whether there is an abnormality in the multiple artificial intelligence chips based on the multiple performance data.
[0217] In some embodiments, each piece of performance data further includes a device identifier of the device to which the corresponding artificial intelligence chip belongs, an operator identifier of the corresponding operator, and communication information. The determination module 1202 is specifically used to:
[0218] Perform communication matching on the operators on the multiple artificial intelligence chips according to the device identifiers, operator identifiers and communication information in the multiple performance data;
[0219] Determine the communication waiting time of each operator pair according to the performance indicator value of each operator pair that is successfully matched;
[0220] Based on the communication waiting time, determine whether there is a communication anomaly in the artificial intelligence chip corresponding to the operator.
[0221] In some embodiments, multiple performance data of each artificial intelligence chip are packaged in at least one message, and the acquisition module 1201 is specifically used to acquire the multiple performance data of the artificial intelligence chip according to the following steps:
[0222] Obtaining from the server the at least one message sent by the artificial intelligence chip;
[0223] The at least one message is unpacked to obtain multiple performance data of the artificial intelligence chip.
[0224] The division of modules in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, the functional modules in the embodiments of the present application may be integrated in a processor, or may exist physically separately, or two or more modules may be integrated in one module. The coupling between the modules may be achieved through some interfaces, which are usually electrical communication interfaces, but it is not excluded that they may be mechanical interfaces or other forms of interfaces. Therefore, the modules described as separate components may or may not be physically separated, and may be located in one place or distributed to different locations of the same or different devices. The above-mentioned integrated modules may be implemented in the form of hardware or in the form of software functional modules.
[0225] An embodiment of the present application also provides an electronic device, including a memory, an artificial intelligence chip, and a computer program stored in the memory and executable on the artificial intelligence chip, wherein the artificial intelligence chip implements the above-mentioned related methods when executing the computer program.
[0226] Next, an electronic device according to another exemplary embodiment of the present application is introduced.
[0227] Refer to the following Fig.13 The electronic device 130 implemented according to this embodiment of the present application is described. Fig.13 The electronic device 130 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.
[0228] like Fig.13 As shown, the electronic device 130 is in the form of a general electronic device. The components of the electronic device 130 may include but are not limited to: the at least one processor 131, the at least one memory 132, and a bus 133 connecting different system components (including the memory 132 and the processor 131).
[0229] Bus 133 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a processor, or a local bus using any of a variety of bus architectures.
[0230] The memory 132 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 1321 and / or a cache memory 1322 , and may further include a read only memory (ROM) 1323 .
[0231] The memory 132 may also include a program / utility 1325 having a set (at least one) of program modules 1324, such program modules 1324 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.
[0232] The electronic device 130 may also communicate with one or more external devices 134 (e.g., keyboards, pointing devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 130, and / or communicate with any device that enables the electronic device 130 to communicate with one or more other electronic devices (e.g., routers, modems, etc.). Such communication may be performed through an input / output (I / O) interface 135. In addition, the electronic device 130 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) through a network adapter 136. As shown, the network adapter 136 communicates with other modules for the electronic device 130 through a bus 133. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 130, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.
[0233] In an exemplary embodiment, a storage medium is also provided, and when a computer program in the storage medium is executed by a processor of an electronic device, the electronic device can perform the above-mentioned data processing method. Optionally, the storage medium can be a non-temporary computer-readable storage medium, for example, the non-temporary computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.
[0234] In an exemplary embodiment, the electronic device of the present application may include at least one processor and a memory communicatively connected to the at least one processor, wherein the memory stores a computer program executable by the at least one processor, and when the computer program is executed by the at least one processor, the at least one processor may execute the steps of any data processing method provided in the embodiments of the present application.
[0235] In an exemplary embodiment, a computer program product is also provided. When the computer program product is executed by an electronic device, the electronic device can implement any exemplary method provided in the present application.
[0236] Furthermore, the computer program product may adopt any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a RAM, a ROM, an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.
[0237] The program product for data processing in the embodiment of the present application may adopt a CD-ROM and include program code, and may be run on a computing device. However, the program product of the present application is not limited thereto, and in this document, a readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, apparatus, or device.
[0238] The readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, wherein the readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.
[0239] The program code contained on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, radio frequency (RF), etc., or any suitable combination of the foregoing.
[0240] Program codes for performing the operations of the present application may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device via any type of network such as a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect via the Internet).
[0241] It should be noted that, although several units or subunits of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into multiple units to be embodied.
[0242] In addition, although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.
[0243] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.
[0244] The present application is described with reference to the flowcharts and / or block diagrams of the methods, apparatus (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0245] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0246] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0247] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.
[0248] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application also includes these modifications and variations.
Claims
1. A data processing method, characterized in that: Applied to artificial intelligence chips, including: In the process of executing the specified code segment of the large model, matching the operator information of each operator corresponding to the specified code segment with at least one operator description information; After determining that the operator information of the operator successfully matches any operator description information, collecting the performance indicator value of the operator according to the performance indicator corresponding to the operator description information, wherein the operator description information and the performance indicator are configured based on the collection control instruction sent by the control end; Based on the performance indicator value of the operator, generate a piece of performance data; sending the performance data to a server; Also includes: In response to the acquisition update instruction sent by the control end, at least one of the operator description information and the performance indicator corresponding to the operator description information is updated.
2. The method according to claim 1, characterized in that After determining that the operator information of the operator successfully matches any operator description information, the method further includes: Matching the calculation timing of the operator with at least one acquisition timing corresponding to the operator description information, wherein the at least one acquisition timing is configured based on the timing control instruction sent by the control terminal; If the calculation timing of the operator successfully matches any collection timing, the performance indicator value of the operator is collected according to the performance indicator corresponding to the operator description information.
3. The method according to claim 2, characterized in that According to the performance indicator corresponding to the operator description information, collecting the performance indicator value of the operator includes: Selecting, from the performance indicators corresponding to the operator description information, a performance indicator corresponding to a successfully matched acquisition opportunity, wherein the performance indicator corresponding to the acquisition opportunity is configured based on the opportunity control instruction; According to the selected performance indicator, the performance indicator value of the operator is collected.
4. The method according to claim 3, characterized in that The acquisition timing is batch information of a training batch, or the acquisition timing is calculation order information in a training batch.
5. The method according to any one of claims 1 to 3, characterized in that: The performance indicator value of the operator includes the execution time and also includes: Storing the start event and the end event of the operator in a record queue; Based on the execution process of the operator, saving the timestamp of the start event and the timestamp of the end event in the record queue; The execution duration of the operator is determined according to the timestamp of the start event and the timestamp of the end event.
6. The method according to claim 5, characterized in that Also includes: Determine the start event and the end event with adjacent timestamps in the record queue as a mergeable event pair; Merge the mergeable event pairs in the record queue into one event.
7. The method according to any one of claims 1 to 3, characterized in that: After determining that the operator information of the operator successfully matches any operator description information, the method further includes: Determine the call relationship information corresponding to the operator through the event stack; Based on the performance indicator value of the operator, a performance data is generated, including: The performance data is generated based on the performance indicator value of the operator and the call relationship information corresponding to the operator.
8. The method according to claim 7, characterized in that The performance data also includes the device identification of the device to which the artificial intelligence chip belongs, the operator identification of the operator, and communication information.
9. The method according to any one of claims 1 to 3, characterized in that: Sending the performance data to a server includes: After the execution of the operator is completed, the performance data stored in the record queue is pushed into the message buffer; After the available space of the message buffer is lower than a preset value or the message buffer meets a clearing condition, packing multiple performance data in the message buffer into one message; The message is sent to the server through the message middleware.
10. The method according to claim 3, characterized in that Also includes: In response to the timing update instruction sent by the control end, at least one of the information of the acquisition timing and the performance indicator corresponding to the acquisition timing is updated.
11. A data processing method, characterized in that: Used in electronic equipment, including: Acquire multiple performance data of multiple artificial intelligence chips from a server, wherein each artificial intelligence chip obtains a piece of performance data according to the following steps: in the process of executing a specified code segment of a large model, match the operator information of each operator corresponding to the specified code segment with at least one operator description information, and after determining that the operator information of the operator successfully matches any operator description information, collect the performance indicator value of the operator according to the performance indicator corresponding to the operator description information, and generate a piece of performance data based on the performance indicator value of the operator, wherein the operator description information and the performance indicator are configured based on the acquisition control instruction sent by the control end, and the artificial intelligence chip is further used to update at least one of the operator description information and the performance indicator in response to the acquisition update instruction sent by the control end; Based on the multiple performance data, determine whether there is any abnormality in the multiple artificial intelligence chips.
12. The method according to claim 11, characterized in that Each piece of performance data also includes a device identifier of the device to which the corresponding artificial intelligence chip belongs, an operator identifier of the corresponding operator, and communication information. Determining whether the plurality of artificial intelligence chips are abnormal based on the plurality of performance data includes: Perform communication matching on the operators on the multiple artificial intelligence chips according to the device identifiers, operator identifiers and communication information in the multiple performance data; Determine the communication waiting time of each operator pair according to the performance indicator value of each operator pair that is successfully matched; Based on the communication waiting time, determine whether there is a communication anomaly in the artificial intelligence chip corresponding to the operator.
13. The method according to claim 11, characterized in that The multiple performance data of each artificial intelligence chip are packaged in at least one message, and the multiple performance data of the artificial intelligence chip are obtained according to the following steps: Obtaining from the server the at least one message sent by the artificial intelligence chip; The at least one message is unpacked to obtain multiple performance data of the artificial intelligence chip.
14. A data processing device, characterized in that: Applied to artificial intelligence chips, including: A matching module, used for matching the operator information of each operator corresponding to the specified code segment with at least one operator description information during the execution of the specified code segment of the large model; A collection module, configured to collect the performance indicator value of the operator according to the performance indicator corresponding to the operator description information after determining that the operator information of the operator successfully matches any operator description information, wherein the operator description information and the performance indicator are configured based on the collection control instruction sent by the control terminal; A generating module, used to generate a piece of performance data based on the performance indicator value of the operator; A sending module, used for sending the performance data to a server; Also includes: An updating module is used to update at least one of the operator description information and the performance indicator corresponding to the operator description information in response to the collection update instruction sent by the control end.
15. A data processing device, characterized in that: Used in electronic equipment, including: An acquisition module, used to acquire multiple performance data of multiple artificial intelligence chips from a server, wherein each artificial intelligence chip obtains a piece of performance data according to the following steps: in the process of executing a specified code segment of a large model, matching the operator information of each operator corresponding to the specified code segment with at least one operator description information, after determining that the operator information of the operator successfully matches any operator description information, collecting the performance indicator value of the operator according to the performance indicator corresponding to the operator description information, and generating a piece of performance data based on the performance indicator value of the operator, wherein the operator description information and the performance indicator are configured based on the acquisition control instruction sent by the control end, and the artificial intelligence chip is further used to update at least one of the operator description information and the performance indicator in response to the acquisition update instruction sent by the control end; A determination module is used to determine whether there are any abnormalities in the multiple artificial intelligence chips based on the multiple performance data.
16. An electronic device comprising a memory, an artificial intelligence chip, and a computer program stored in the memory and executable on the artificial intelligence chip, characterized in that: When the artificial intelligence chip executes the computer program, it implements the method described in any one of claims 1 to 10.
17. An electronic device, characterized in that: include: at least one processor, and a memory communicatively connected to the at least one processor, wherein: The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the method according to any one of claims 11 to 13.
18. A storage medium, characterized in that: When the computer program in the storage medium is executed by a processor of an electronic device, the electronic device can execute the method according to any one of claims 1 to 13.
19. A computer program product, characterized in that The method comprises a computer program, which implements the method according to any one of claims 1 to 13 when executed by a processor.
