Slave node performance analysis method and device, electronic equipment and storage medium

Abstract

The invention discloses a slave node performance analysis method and device, electronic equipment and a storage medium, and relates to the technical field of computers, the method comprises the following steps: obtaining an inference task, a first model code and a plurality of timestamp recorders; splitting the reasoning task into N sub-tasks, and determining M slave nodes corresponding to execution of the N sub-tasks; inserting a plurality of timestamp recorders into a plurality of key positions in the first model code to generate a new second model code; timestamp information and calling function information, recorded by each timestamp recorder, of each slave node when each subtask is executed are obtained, and multiple pieces of timestamp information and multiple pieces of calling function information are obtained; and generating a corresponding performance state table based on the multiple pieces of timestamp information and the multiple pieces of calling function information, and determining the performance states of the M slave nodes according to the performance state table. The problem that the performance analysis of a single slave node is limited, and the reasoning task execution conditions of a plurality of slave nodes cannot be accurately determined is solved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a performance analysis method, device, electronic device, and storage medium for a slave node. Background Art

[0002] As models have grown exponentially in size and complexity, from initially small neural networks with a few million parameters to today's massive models with hundreds of billions or even trillions of parameters, they have demonstrated remarkable capabilities in fields such as natural language processing, computer vision, and speech recognition. However, as models scale, their inference performance has become a significant bottleneck. In practical applications, latency during the inference phase directly impacts user experience and system efficiency. Model inference typically requires a master node to send inference tasks to multiple slave nodes for execution. Therefore, the performance of each slave node impacts the overall latency of the inference phase. To optimize the model's inference performance, it's necessary to analyze the performance of each slave node.

[0003] Slave node performance analysis is typically performed using traditional performance analysis tools (such as Nsight Systems and the PyTorch Profiler). However, these tools typically only analyze the performance of a single slave node and typically display performance data at the kernel level, without being able to directly link it to specific upper-level code locations. This makes it difficult to accurately determine the execution status of inference tasks across multiple slave nodes. Summary of the Invention

[0004] The present application provides a performance analysis method, device, electronic device and storage medium for a slave node, so as to at least solve the problem in the related art that the performance analysis of a single slave node is limited and the execution status of reasoning tasks of multiple slave nodes cannot be accurately determined.

[0005] The present application provides a performance analysis method for a slave node, which is applied to a master node. The method comprises: obtaining an inference task, a first model code, and a plurality of timestamp recorders; splitting the inference task into N subtasks, and determining M slave nodes corresponding to the N subtasks, wherein each subtask is executed by calling a function by at least one slave node, N≥2, M≥1; determining a plurality of key positions in the first model code, and inserting a plurality of timestamp recorders into the plurality of key positions to generate a new second model code, wherein the second model code provides a function to be called for each slave node to execute each subtask; obtaining the timestamps recorded by each timestamp recorder; The timestamp information and calling function information of each slave node when executing each subtask are obtained, and multiple timestamp information and multiple calling function information of M slave nodes are obtained. The timestamp information includes the start and end timestamps of the master node submitting each subtask to each slave node, and the start and end timestamps of each slave node executing each subtask. The calling function information includes the code line number and function name of a function that needs to be called for each subtask in the second model code, as well as the node identifier of each slave node; based on the multiple timestamp information and multiple calling function information, a performance status table corresponding to the M slave nodes is generated, and the performance status of the M slave nodes is determined according to the performance status table.

[0006] In addition, the present application also provides a performance analysis device for a slave node, the device comprising:

[0007] An acquisition module, configured to acquire an inference task, a first model code, and a plurality of timestamp recorders;

[0008] A processing module is configured to split the inference task into N subtasks and determine M slave nodes corresponding to the N subtasks, where each subtask is executed by calling a function by at least one slave node, and N ≥ 2 and M ≥ 1;

[0009] The processing module is further configured to determine a plurality of key positions in the first model code, insert a plurality of time stamp recorders into the plurality of key positions, and generate a new second model code, wherein the second model code provides a function to be called for each slave node to execute each subtask;

[0010] The acquisition module is further used to obtain the timestamp information and call function information of each slave node recorded by each timestamp recorder when executing each subtask, and obtain multiple timestamp information and multiple call function information of M slave nodes, the timestamp information includes the start and end timestamps of the master node submitting each subtask to each slave node, and the start and end timestamps of each slave node executing each subtask, and the call function information includes the code line number and function name of a function that needs to be called for each subtask in the second model code, and the node identifier of each slave node;

[0011] The processing module is further configured to generate a performance status table corresponding to the M slave nodes based on multiple timestamp information and multiple call function information, and determine the performance status of the M slave nodes according to the performance status table.

[0012] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-mentioned slave node performance analysis methods when executing the computer program.

[0013] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned slave node performance analysis method are implemented.

[0014] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned slave node performance analysis method when executed by a processor.

[0015] The performance analysis method, device and electronic device of the slave node provided in the present application add the start and end timestamps of each subtask submitted by the master node to each slave node, as well as the start and end timestamps of each subtask executed by each slave node in the timestamp information. The call time of the master node can be accurately determined by using the start and end timestamps of each subtask submitted by the master node to each slave node, and the actual execution time of the slave node can be accurately determined by using the start and end timestamps of each subtask executed by the slave node. That is, the execution time of the master node and the slave node can be accurately distinguished by comparing the call time of the master node and the actual execution time of the slave node.

[0016] In addition, the performance status table also includes the execution time (time consumption) of each subtask executed by each slave node in multiple slave nodes. Based on the performance status table, the real-time synchronization and summary analysis of the timestamp data of each node in the multiple slave node environment, as well as the code call link and time distribution of each slave node calling function can be analyzed, thereby accurately determining the execution status of the reasoning tasks of multiple slave nodes, and matching the performance data with the specific code line number, so that researchers can quickly locate abnormal code on the upper-level code side. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0018] Figure 1 A topological diagram of a performance analysis system for a slave node provided in an embodiment of the present application;

[0019] Figure 2 A schematic diagram of a flow chart of a performance analysis method for a slave node provided in an embodiment of the present application;

[0020] Figure 3 A structural block diagram of a performance analysis device for a slave node provided in an embodiment of the present application;

[0021] Figure 4 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0022] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0023] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0024] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0025] In conjunction with the specific application environment architecture or specific hardware architecture on which the execution of the performance analysis method of the slave node depends, the specific application environment architecture or specific hardware architecture is described herein.

[0026] The embodiments of the present application are applied to scenarios where multiple slave nodes perform reasoning tasks during model operation.

[0027] In related technologies, in order to optimize the inference performance of the model, researchers need to conduct in-depth analysis of the time distribution during the model operation process. Specifically, they need to understand the time sequence of the model calling code, the task allocation between the master node and the slave node, the execution status of asynchronous operations, and the synchronization overhead in a multi-card environment (multiple slave nodes). Although traditional performance analysis tools can provide a certain degree of performance data, they often have single-card limitations and cannot monitor the execution of multiple slave nodes at the same time. In addition, traditional performance analysis tools lack code context and usually display performance data at the kernel level granularity, and cannot be directly associated with specific upper-level code locations. The reports generated by traditional performance analysis tools often contain a large amount of underlying hardware performance information such as the execution time and memory transfer when each core in a single slave node performs inference tasks, which is difficult for non-professionals to interpret.

[0028] In order to solve the above technical problems, an embodiment of the present application provides a performance analysis method for a slave node. This method can insert a lightweight timestamp recorder to dynamically obtain the execution reasoning task time of the master node and each slave node during the model reasoning process, and then accurately determine the problem of the execution status of the reasoning tasks of multiple slave nodes. It also supports simultaneous performance analysis of multiple slave nodes and corresponds the performance data with specific code line numbers, so that researchers can quickly locate abnormal code on the upper-level code end.

[0029] The method provided in the embodiment of the present application is described below using the performance analysis system of the slave node as an example.

[0030] like Figure 1 As shown, Figure 1 A topological diagram of a performance analysis system for a slave node provided in an embodiment of the present application. Figure 1 In the embodiment, the performance analysis system 100 of the slave nodes includes: a master node 101, a first slave node 102 and a second slave node 103. The master node 101 is connected to each of the slave nodes via a communication interface.

[0031] The master node 101 (also referred to as the performance analysis device 101 of the slave node) can be any device with computing and communication capabilities. For example, the master node can be a central processing unit (CPU) node, also referred to as a CPU node. Optionally, the master node 101 also has a display interface or screen.

[0032] It is understood that the master node's operating system has necessary software dependencies installed, including but not limited to programming language software (such as Python), deep learning frameworks (such as PyTorch or TensorFlow), and the Compute Unified Device Architecture (CUDA) toolkit. The master node is also configured with a global parameter file or command line input interface. This interface is used to transmit the user's configuration items for performance analysis of multiple slave nodes set on the display interface. Among them, the configuration items may include whether to enable multi-card mode and the output path of the performance status table.

[0033] The first slave node 102 and the second slave node 103 may be any device with computing and communication capabilities. For example, the first slave node 102 or the second slave node 103 may be a graphics processing unit (GPU) node, also known as a GPU node or a GPU card.

[0034] Furthermore, it should be understood that Figure 1 In the example shown in FIG, only two GPU slave nodes are shown. In fact, the system may also include more or fewer GPU nodes, and this embodiment does not limit this.

[0035] The embodiment of the present application provides a performance analysis method for a slave node. Figure 1 The system shown in FIG. 1 can further be applied to the master node 101 in the system, such as Figure 2 As shown, Figure 2 A schematic flow chart of a performance analysis method for a slave node provided in an embodiment of the present application, the performance analysis method for a slave node includes the following steps:

[0036] S201: Acquire an inference task, a first model code, and multiple timestamp recorders.

[0037] The inference task needs to call the first model code for execution. The first model code includes multiple functions.

[0038] The timestamp recorder is used to record the current timestamp when entering or exiting different functions.

[0039] Exemplarily, one implementation of obtaining the inference task and the first model code in step S201 is as follows: the master node obtains the inference task and the first model code in response to a user input or selection operation on a display interface. The input or selection operation may be an operation in which the user inputs or selects a storage path for the inference task and the first model code.

[0040] Optionally, before step S201, it also includes: starting the global performance recording module included in the master node, initializing a shared queue through the global performance recording module, and registering multiple timestamp generation functions to obtain multiple timestamp recorders. At the same time, configuring the communication channel between each slave node (for example, by using the distributed communication protocol (NVIDIA Collective Communications Library, NCCL) to establish a communication channel with each slave node) to ensure that the data of each slave node can be synchronized to the master node in real time.

[0041] S202: Split the inference task into N subtasks, and determine M slave nodes corresponding to executing the N subtasks.

[0042] In one example, the master node splits the inference task into N subtasks according to the N functions included in the model code required for the inference task.

[0043] Each subtask is executed by calling a function by at least one slave node, N ≥ 2, M ≥ 1.

[0044] It is understood that each subtask corresponds to a function, and each subtask can be executed by one or more slave nodes. A slave node can execute multiple subtasks.

[0045] S203: Determine multiple key positions in the first model code, and insert multiple time stamp recorders into the multiple key positions to generate a new second model code.

[0046] The second model code provides a function to be called for each slave node to execute each subtask.

[0047] Key locations may include the backbone structure boundary operation function locations of the model code, the data transmission operation function locations between the master node and the slave node, the collective communication operation function locations between multiple slave nodes, and the matrix multiplication or convolution calculation-intensive operator operation function locations.

[0048] The boundary operation functions of the model code's backbone structure can be located at the entry and exit points of the core modules included in the model code. Inserting a timestamp recorder at this location can record the overall execution time of each core code module (such as TransformerBlock), allowing for quick identification of code modules with performance anomalies (for example, discovering that a particular attention layer consumes too much time).

[0049] Inserting a timestamp recorder at the data transfer operation function between the master node and the slave node can record the data transfer time and memory changes, so as to identify unnecessary data transfer or memory bottlenecks.

[0050] Inserting a timestamp recorder at the collective communication operation function (such as AllReduce function and AllGather function) between multiple slave nodes can record the communication time and synchronization waiting time, so as to detect multi-card load imbalance or network bottleneck.

[0051] Inserting a timestamp recorder at the location of matrix multiplication or convolution calculation-intensive operator operation functions can record the computational time and hardware utilization, so as to locate the computational hotspots and evaluate the operator optimization effect.

[0052] It can be understood that multiple key positions are the positions of multiple key functions or important functions.

[0053] In one example, after determining multiple key locations in the first model code, the master node uses a decorator or a context manager to insert multiple timestamp recorders into the multiple key locations to generate a new second model code.

[0054] It can be understood that each time a decorated function (i.e., a function into which a timestamp recorder is inserted) is entered or exited, the current timestamp is automatically recorded and pushed into a shared queue together with the function name and the line number of the code where the function is located.

[0055] S204: Acquire the timestamp information and the calling function information of each slave node when executing each subtask recorded by each timestamp recorder, and obtain multiple timestamp information and multiple calling function information of M slave nodes.

[0056] Timestamp information includes the start and end timestamps of each subtask submitted by the master node to each slave node, as well as the start and end timestamps of each subtask executed by each slave node. These timestamps include a start timestamp and an end timestamp. The end timestamp can be recorded after each subtask completes using the CUDA stream callback mechanism. CUDA streams are used to manage the parallel execution of different computing tasks on slave nodes.

[0057] For example, Figure 1 For example, when the master node 101 begins submitting a subtask to the first slave node 102, it records a start timestamp and an end timestamp after the submission is complete. Subsequently, after receiving the subtask, the slave node records a start timestamp when it begins executing the subtask and, after completing the subtask, uses the CUDA stream callback mechanism to record an end timestamp.

[0058] The calling function information includes the code line number and function name of a function that each subtask needs to call in the second model code, and the node identifier of each slave node.

[0059] Optionally, the subsequent master node may organize the timestamp information from each slave node into a pre-set format that is easy to parse, for example, a JSON file format.

[0060] Understandably, when the master node submits a subtask to a slave node, the slave node may be executing other subtasks. That is, the slave node may not start executing the subtask immediately after receiving it. Therefore, by monitoring the state changes of each task (CUDA Event) in the CUDA stream, the start and end times of the asynchronous task are accurately captured, and the start and end timestamps are recorded. After each slave node executes a subtask, the multiple timestamps recorded locally are sent to the master node.

[0061] In some optional embodiments, the master node submits N subtasks to M slave nodes, and based on multiple timestamp recorders, records the start and end timestamps and calling function information of each subtask submitted by the master node to each slave node; receives the start and end timestamps of each slave node executing each subtask sent by the M slave nodes, and based on the start and end timestamps of each subtask submitted by the master node to each slave node, the start and end timestamps of each slave node executing each subtask, and the calling function information, obtains multiple timestamp information and multiple calling function information of the M slave nodes.

[0062] The start and end timestamps of each slave node executing each subtask are recorded by each slave node calling each function to execute each subtask and recorded by each timestamp.

[0063] The following describes in detail the specific execution method of obtaining multiple timestamp information and multiple call function information of M slave nodes in two different scenarios.

[0064] Scenario 1: N subtasks include the first subtask. The first subtask requires L slave nodes to call the first function in the second model code to execute, 1<L≤M.

[0065] In one example, a master node submits a first subtask to L slave nodes, and based on multiple timestamp recorders, records the first start and end timestamps and first calling function information of the master node submitting the first subtask to each of the L slave nodes; receives the second start and end timestamps sent by each of the L slave nodes; based on the first start and end timestamps, the second start and end timestamps and the first calling function information, obtains multiple timestamp information and multiple calling function information of each of the L slave nodes executing the first subtask.

[0066] Each first start and end timestamp is a start and end timestamp for the master node to submit the first subtask to each slave node.

[0067] The first calling function information includes each node identifier of each slave node, a first function name of the first function, and a first code line number of the first function in the second model code.

[0068] Each second start and end timestamp is a start and end timestamp of each slave node calling the first function to execute the first subtask.

[0069] Scenario 2: Multiple slave nodes include a first slave node; N subtasks further include P subtasks, and each of the P subtasks needs to call one of the P second functions included in the model code, 1<P≤N.

[0070] In one example, the master node submits each of P subtasks to the first slave node, and based on multiple timestamp recorders, records the third start and end timestamps and second calling function information of the master node submitting each of the P subtasks to the first slave node; receives multiple fourth start and end timestamps sent by the first slave node; and based on the third start and end timestamps, the fourth start and end timestamps and the second calling function information, obtains multiple timestamp information and multiple calling function information of the timestamp information of the first slave node executing the P subtasks.

[0071] Each third start and end timestamp is the start and end timestamp of each subtask in the P subtasks submitted by the master node.

[0072] The second calling function information includes the first node identifier of the first slave node, each second function name of each second function, and each second code line number of each second function in the second model code.

[0073] Each fourth start and end timestamp includes a start and end timestamp of each second function called by the first slave node to execute each subtask.

[0074] Understandably, since the deep learning framework allows the mixed use of CPU resources and GPU resources, it is necessary to clearly distinguish and record the different start and end timestamps of the master node and the slave node.

[0075] S205: Based on the multiple timestamp information and the multiple call function information, generate a performance status table corresponding to the M slave nodes, and determine the performance status of the M slave nodes according to the performance status table.

[0076] The performance status table records the time consumption information required for each slave node to execute each corresponding subtask and the function information of the function called when executing the subtask.

[0077] In some optional embodiments, the master node determines each first call time of each of the multiple slave nodes based on the first start and end timestamps; determines each first execution time of each of the multiple slave nodes based on the second start and end timestamps; establishes a first mapping relationship between the first function name, the first code line number and each node identifier, each first call time, and each first execution time of each slave node in the L slave nodes; determines each second call time of each of the P subtasks based on the third start and end timestamps; determines each second execution time of each of the P subtasks based on the fourth start and end timestamps; establishes a second mapping relationship between the first node identifier and each second function name, each second call time, each second execution time, and each second code line number, and generates performance status tables corresponding to the M slave nodes based on the first mapping relationship and the second mapping relationship.

[0078] Each first calling time is used to indicate the time taken by the master node to submit the first subtask to each slave node.

[0079] Each first execution time is used to indicate the time taken by each slave node to execute the first subtask.

[0080] Each second call time is used to indicate the time taken by the master node to submit each subtask to each slave node.

[0081] Each second execution time is used to indicate the time taken by the first slave node to call each second function to execute each subtask.

[0082] In one example, as shown in Table 1, Table 1 is a performance status table.

[0083] Table 1

[0084]

[0085] In some optional embodiments, the plurality of slave nodes include a second slave node and a third slave node. The master node obtains, from the performance status table, a first target execution time corresponding to the target function name corresponding to the node identifier of the second slave node, and obtains a second target execution time corresponding to the target function name corresponding to the node identifier of the third slave node; compares the first target execution time with the second target execution time; and if the first target execution time is greater than the second target execution time, determines a cause of the abnormality of the second slave node based on the target function name and the first preset correspondence.

[0086] The first preset corresponding relationship is the corresponding relationship between the target function name and the abnormality cause.

[0087] For example, taking Table 1 as an example, the master node obtains from the performance status table a first target execution time of 58ms corresponding to the target function name MatMul corresponding to slave node 0, and obtains a second target execution time of 42ms corresponding to the target function name corresponding to the node identifier of slave node 1; comparing the first target execution time and the second target execution time, the first target execution time is greater than the second target execution time, and based on the target function name MatMul and the first preset correspondence, it is determined that the cause of the abnormality of slave node 0 is the limited computing bandwidth of slave node 0. The first preset correspondence is the correspondence between MatMul and the cause of the abnormality being limited computing bandwidth.

[0088] For another example, the master node obtains from the performance status table the first target execution time corresponding to the target function name AllReduce corresponding to slave node 0, which is 15ms, and obtains the second target execution time corresponding to the target function name corresponding to the node identifier of slave node 1, which is 28ms. The master node compares the first target execution time and the second target execution time. The first target execution time is less than the second target execution time. Based on the target function name MatMul and the first preset correspondence, the master node determines that the cause of the abnormality in slave node 1 is high network latency in slave node 1. The first preset correspondence is the correspondence between AllReduce and the cause of the abnormality being high network latency.

[0089] In some optional embodiments, the plurality of slave nodes include a fourth slave node. The master node obtains, from the performance status table, a plurality of third target execution times corresponding to all function names corresponding to the node identifier of the fourth slave node; determines a maximum third target execution time from the plurality of third target execution times; searches the performance status table for a target code line number corresponding to the maximum third target execution time, and generates prompt information including the target code line number.

[0090] The prompt information is used to indicate the function at the position corresponding to the target code line number in the model code that needs to be optimized.

[0091] Furthermore, the master node obtains the total time spent on the inference task, calculates the ratio of the execution time of each subtask on each slave node to the total time, and obtains the percentage of the time consumed by the function corresponding to each subtask in the total inference task time. The master node then records and displays the percentages, which reflect the time consumption of each function or subtask.

[0092] It is understandable that the user can determine the resource consumption proportion of each subtask or each function by viewing multiple percentages.

[0093] Optionally, the master node obtains each execution time and each corresponding call time of each slave node in multiple slave nodes; calculates the sum of each execution time and each call time to obtain each first duration for each slave node to complete the subtask; calculates the ratio of each execution time to each first duration to obtain the first resource utilization (GPU resource utilization) for each slave node to complete the subtask; calculates the ratio of each call time to each first duration to obtain the second resource utilization (CPU resource utilization) for each slave node to complete the subtask; when the first resource utilization is less than the first value and the second resource utilization is greater than the second value, the number of tasks of each slave node is increased according to the adjustment ratio of each slave node; when the first resource utilization is greater than or equal to the first value, or the second resource utilization is less than or equal to the second value, the number of tasks of each slave node is reduced according to the adjustment ratio of each slave node.

[0094] The adjustment ratio is determined based on a preset formula, which may be:

[0095]

[0096] Among them, δ represents the adjustment ratio of each slave node, U CPU Indicates the first resource utilization. U GPU Indicates the second resource utilization.

[0097] Based on the above Figure 2 The method shown can obtain an inference task, a first model code, and multiple timestamp recorders; split the inference task into N subtasks, and determine the M slave nodes corresponding to the N subtasks; and insert multiple timestamp recorders into multiple key positions in the first model code, so as to record the timestamp information and calling function information of each slave node when executing each subtask through each timestamp recorder, and generate a corresponding performance status table based on the timestamp information and calling function information, and determine the performance status of the M slave nodes according to the performance status table.

[0098] Since the timestamp information includes the start and end timestamps of each subtask submitted by the master node to each slave node, as well as the start and end timestamps of each subtask executed by each slave node, the execution time of the master node and the slave node can be accurately distinguished, and then the master node call time and the actual execution time of the slave node can be compared to accurately calculate the specific time consumption on the hardware. In addition, since the performance status table includes the execution time (time consumption) of each slave node in the multiple slave nodes to execute each subtask, the timestamp data of each node in the multiple slave node environment can be synchronized and summarized in real time based on the performance status table, as well as the code call link and time distribution of each slave node calling the function, and then the execution status of the reasoning tasks of multiple slave nodes can be accurately determined. The performance data is matched with the specific line number of the code, so that researchers can quickly locate abnormal code on the upper code side.

[0099] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0100] The embodiment of the present application also provides a performance analysis device for a slave node, such as Figure 3 As shown, Figure 3 This is a structural block diagram of a performance analysis device for a slave node provided in an embodiment of the present application; the performance analysis device for a slave node includes:

[0101] An acquisition module 301 is configured to acquire an inference task, a first model code, and a plurality of time stamp recorders;

[0102] Processing module 302 is configured to split the inference task into N subtasks and determine M slave nodes corresponding to the N subtasks, wherein each subtask is executed by calling a function by at least one slave node, N ≥ 2, M ≥ 1;

[0103] The processing module 302 is further configured to determine a plurality of key locations in the first model code, insert a plurality of timestamp recorders into the plurality of key locations, and generate a new second model code, wherein the second model code provides a function to be called for each slave node to execute each subtask;

[0104] The acquisition module 301 is further configured to acquire the timestamp information and the calling function information of each slave node when executing each subtask, as recorded by each timestamp recorder, to obtain multiple timestamp information and multiple calling function information of the M slave nodes, wherein the timestamp information includes the start and end timestamps of the master node submitting each subtask to each slave node, and the start and end timestamps of each slave node executing each subtask; the calling function information includes the code line number and function name of a function to be called by each subtask in the second model code, and the node identifier of each slave node;

[0105] The processing module 302 is further configured to generate a performance status table corresponding to the M slave nodes based on the multiple timestamp information and the multiple call function information, and determine the performance status of the M slave nodes according to the performance status table.

[0106] In some optional embodiments, the acquisition module 301 is specifically used to submit N subtasks to M slave nodes, and based on multiple timestamp recorders, record the start and end timestamps and calling function information of each subtask submitted by the master node to each slave node; receive the start and end timestamps of each slave node executing each subtask sent by the M slave nodes, the start and end timestamps of each slave node executing each subtask are each slave node calling each function to execute each subtask and recorded by each timestamp, based on the start and end timestamps of each subtask submitted by the master node to each slave node, the start and end timestamps of each slave node executing each subtask and the calling function information, to obtain multiple timestamp information and multiple calling function information of the M slave nodes.

[0107] In some optional embodiments, the N subtasks include a first subtask, and the first subtask requires L slave nodes to call the first function in the second model code to execute, 1<L≤M; the acquisition module 301 is also specifically used to submit the first subtask to the L slave nodes, and based on multiple timestamp recorders, record the first start and end timestamps and first calling function information of the master node submitting the first subtask to each slave node in the L slave nodes, each first start and end timestamp is the start and end timestamp of the master node submitting the first subtask to each slave node, and the first calling function information includes each node identifier of each slave node, the first function name of the first function, and the first code line number of the first function in the second model code; receive the second start and end timestamps sent by each slave node in the L slave nodes, each second start and end timestamp is the start and end timestamp of each slave node calling the first function to execute the first subtask; based on the first start and end timestamps, the second start and end timestamps and the first calling function information, obtain multiple timestamp information and multiple calling function information of each slave node in the L slave nodes executing the first subtask.

[0108] In some optional embodiments, the multiple slave nodes include a first slave node; the N subtasks also include P subtasks, each of the P subtasks needs to call one of the P second functions included in the model code, 1<P≤N; the acquisition module 301 is further specifically used to submit each of the P subtasks to the first slave node, and based on multiple timestamp recorders, record the third start and end timestamps and second calling function information of the master node submitting each of the P subtasks to the first slave node, each third start and end timestamp is the start and end timestamp of the master node submitting each of the P subtasks, and the second calling function information includes the first node identifier of the first slave node, each second function name of each second function, and each second code line number of each second function in the second model code; receive multiple fourth start and end timestamps sent by the first slave node, each fourth start and end timestamp includes the start and end timestamps of the first slave node calling each second function to execute each subtask; based on the third start and end timestamps, the fourth start and end timestamps and the second calling function information, obtain multiple timestamp information and multiple calling function information of the timestamp information of the first slave node executing the P subtasks.

[0109] In some optional embodiments, the processing module 302 is specifically used to determine each first call time of each of the multiple slave nodes based on the first start and end timestamps, each first call time is used to indicate the time taken by the master node to submit the first subtask to each slave node; determine each first execution time of each of the multiple slave nodes based on the second start and end timestamps, each first execution time is used to indicate the time taken by each slave node to execute the first subtask; establish a first mapping relationship between the first function name, the first code line number and each node identifier of each slave node in the L slave nodes, each first call time, and each first execution time; determine each second call time of each of the P subtasks based on the third start and end timestamps, each second call time is used to indicate the time taken by the master node to submit each subtask to each slave node; determine each second execution time of each of the P subtasks based on the fourth start and end timestamps, each second execution time is used to indicate the time taken by the first slave node to call each second function to execute each subtask; establish a second mapping relationship between the first node identifier and each second function name, each second call time, each second execution time, and each second code line number. According to the first mapping relationship and the second mapping relationship, performance status tables corresponding to the M slave nodes are generated.

[0110] In some optional embodiments, the processing module 302 is further specifically used to obtain the first target execution time corresponding to the target function name corresponding to the node identifier of the second slave node from the performance status table, and to obtain the second target execution time corresponding to the target function name corresponding to the node identifier of the third slave node; compare the first target execution time and the second target execution time; if the first target execution time is greater than the second target execution time, determine the abnormal cause of the second slave node based on the target function name and the first preset correspondence, and the first preset correspondence is the correspondence between the target function name and the abnormal cause.

[0111] In some optional embodiments, the multiple slave nodes include a fourth slave node; the acquisition module 301 is also used to obtain multiple third target execution times corresponding to all function names corresponding to the node identifier of the fourth slave node from the performance status table; the processing module 302 is also used to determine the maximum third target execution time from the multiple third target execution times; the processing module 302 is also used to search the target code line number corresponding to the maximum third target execution time from the performance status table, and generate prompt information including the target code line number, the prompt information is used to indicate the function corresponding to the target code line number in the model code that needs to be optimized.

[0112] For the description of the features in the embodiment corresponding to the performance analysis device of the slave node, reference can be made to the relevant description of the embodiment corresponding to the performance analysis method of the slave node, which will not be repeated here.

[0113] The embodiment of the present application also provides an electronic device, such as Figure 4 As shown, Figure 4 This is a hardware structure diagram of an electronic device provided in an embodiment of the present application. The electronic device can Figure 1 The performance analysis device 101 of the slave node shown is an electronic device comprising a processor 10 and a memory 20, wherein the memory 20 stores a computer program, and the processor 10 is configured to run the computer program to execute the steps in any of the above-mentioned slave node performance analysis method embodiments.

[0114] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above-mentioned slave node performance analysis method embodiments when running.

[0115] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0116] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any of the above-mentioned embodiments of the performance analysis method for a slave node are implemented.

[0117] An embodiment of the present application also provides another computer program product, including a non-volatile computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps in any of the above-mentioned slave node performance analysis method embodiments.

[0118] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0119] The above is a detailed introduction to the performance analysis method, device, electronic device and storage medium of the slave node provided by this application. Specific examples are used herein to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core idea of this application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of this application, several improvements and modifications can be made to this application, and these improvements and modifications also fall within the scope of protection of the claims of this application.

Claims

1. A performance analysis method for a slave node, characterized in that: Applied to the master node, the method includes: Obtaining an inference task, a first model code, and a plurality of timestamp recorders; Splitting the inference task into N subtasks, and determining M slave nodes corresponding to the N subtasks, wherein each subtask is executed by calling a function by at least one slave node, N≥2, M≥1; Determining a plurality of key positions in the first model code, and inserting the plurality of timestamp recorders into the plurality of key positions, to generate a new second model code, wherein the second model code provides a function to be called for each slave node to execute each subtask; Obtaining timestamp information and calling function information recorded by each timestamp recorder when each slave node executes each subtask, to obtain multiple timestamp information and multiple calling function information of M slave nodes, wherein the timestamp information includes the start and end timestamps of the master node submitting each subtask to each slave node, and the start and end timestamps of each slave node executing each subtask, and the calling function information includes the code line number and function name of a function that needs to be called by each subtask in the second model code, and the node identifier of each slave node; Based on the multiple timestamp information and the multiple call function information, a performance status table corresponding to the M slave nodes is generated, and the performance status of the M slave nodes is determined according to the performance status table.

2. The method according to claim 1, characterized in that The acquiring of the timestamp information and the calling function information of each slave node when executing each subtask recorded by each timestamp recorder to obtain multiple timestamp information and multiple calling function information of M slave nodes includes: Submitting the N subtasks to the M slave nodes, and recording the start and end timestamps and call function information of each subtask submitted by the master node to each slave node based on the multiple timestamp recorders; Receive the start and end timestamps of each slave node executing each subtask sent by the M slave nodes, where the start and end timestamps of each slave node executing each subtask are recorded by calling each function of each slave node to execute each subtask and recording each timestamp. Based on the start and end timestamps of each subtask submitted by the master node to each slave node, the start and end timestamps of each slave node executing each subtask and the calling function information, the multiple timestamp information and the multiple calling function information of the M slave nodes are obtained.

3. The method according to claim 2, characterized in that The N subtasks include a first subtask, where the first subtask requires L slave nodes to call a first function in the second model code for execution, where 1<L≤M; The acquiring of the timestamp information and the calling function information of each slave node when executing each subtask recorded by each timestamp recorder to obtain multiple timestamp information and multiple calling function information of M slave nodes includes: Submitting the first subtask to the L slave nodes, and recording, based on the multiple timestamp recorders, first start and end timestamps and first calling function information of the master node submitting the first subtask to each of the L slave nodes, each first start and end timestamp being the start and end timestamp of the master node submitting the first subtask to each slave node, and the first calling function information including each node identifier of each slave node, a first function name of the first function, and a first code line number of the first function in the second model code; Receiving a second start and end timestamp sent by each of the L slave nodes, where each of the second start and end timestamps is a start and end timestamp for each slave node to call the first function to execute the first subtask; Based on the first start and end timestamps, the second start and end timestamps and the first calling function information, multiple timestamp information and multiple calling function information of each of the L slave nodes executing the first subtask are obtained.

4. The method according to claim 3, characterized in that The multiple slave nodes include a first slave node; the N subtasks further include P subtasks, each of the P subtasks needs to call one of the P second functions included in the model code, 1<P≤N; the acquiring of timestamp information and calling function information recorded by each timestamp recorder when each slave node executes each subtask to obtain multiple timestamp information and multiple calling function information of the M slave nodes includes: Submit each of the P subtasks to the first slave node, and record, based on the multiple timestamp recorders, third start and end timestamps and second calling function information of the master node submitting each of the P subtasks to the first slave node, each of the third start and end timestamps being the start and end timestamps of the master node submitting each of the P subtasks, and the second calling function information including a first node identifier of the first slave node, each second function name of each of the second functions, and each second code line number of each of the second functions in the second model code; receiving a plurality of fourth start and end timestamps sent by the first slave node, each of the fourth start and end timestamps including a start and end timestamp of each second function called by the first slave node to execute each subtask; Based on the third start and end timestamps, the fourth start and end timestamps and the second calling function information, multiple timestamp information and multiple calling function information of the timestamp information of the first slave node executing the P subtasks are obtained.

5. The method according to claim 4, characterized in that The generating, based on the multiple timestamp information and the multiple call function information, the performance status tables corresponding to the M slave nodes, includes: Determine, according to the first start and end timestamps, each first calling time of each of the plurality of slave nodes, wherein each first calling time is used to indicate the time taken by the master node to submit the first subtask to each of the slave nodes; determining, based on the second start and end timestamps, each first execution time of each of the plurality of slave nodes, wherein each first execution time is used to indicate the time taken by each of the slave nodes to execute the first subtask; and establishing a first mapping relationship between the first function name, the first code line number, and each node identifier of each of the L slave nodes, each first call time, and each first execution time; Determine, according to the third start and end timestamps, each second call time of each of the P subtasks, where each second call time is used to indicate the time taken by the master node to submit each of the subtasks to each of the slave nodes; Determine, according to the fourth start and end timestamps, each second execution time of each of the P subtasks, where each second execution time is used to indicate the time taken by the first slave node to call each second function to execute each subtask; Establishing a second mapping relationship between the first node identifier and each second function name, each second call time, each second execution time, and each second code line number; A performance status table corresponding to the M slave nodes is generated according to the first mapping relationship and the second mapping relationship.

6. The method according to claim 5, characterized in that The multiple slave nodes include a second slave node and a third slave node; and determining the performance status of the M slave nodes according to the performance status table includes: Obtaining from the performance status table a first target execution time corresponding to the target function name corresponding to the node identifier of the second slave node, and obtaining a second target execution time corresponding to the target function name corresponding to the node identifier of the third slave node; comparing the first target execution time and the second target execution time; If the first target execution time is greater than the second target execution time, the abnormal cause of the second slave node is determined based on the target function name and a first preset correspondence, where the first preset correspondence is the correspondence between the target function name and the abnormal cause.

7. The method according to claim 6, characterized in that The plurality of slave nodes include a fourth slave node; the method further comprising: Acquire, from the performance status table, a plurality of third target execution times corresponding to all function names corresponding to the node identifier of the fourth slave node; determining a maximum third target execution time from the plurality of third target execution times; The target code line number corresponding to the maximum third target execution time is searched from the performance status table, and prompt information including the target code line number is generated, wherein the prompt information is used to indicate that the function at the position corresponding to the target code line number in the model code needs to be optimized.

8. A performance analysis device for a slave node, characterized in that: Applied to a master node, the device comprises: An acquisition module, configured to acquire an inference task, a first model code, and a plurality of timestamp recorders; a processing module, configured to split the inference task into N subtasks, and determine M slave nodes corresponding to the N subtasks, wherein each subtask is executed by calling a function by at least one slave node, N ≥ 2, M ≥ 1; The processing module is further configured to determine a plurality of key positions in the first model code, and insert the plurality of timestamp recorders into the plurality of key positions to generate a new second model code, wherein the second model code provides a function to be called for each slave node to execute each subtask; The acquisition module is further configured to acquire the timestamp information and the calling function information of each slave node when executing each subtask, as recorded by each timestamp recorder, to obtain multiple timestamp information and multiple calling function information of the M slave nodes, wherein the timestamp information includes the start and end timestamps of the master node submitting each subtask to each slave node, and the start and end timestamps of each slave node executing each subtask, and the calling function information includes the code line number and function name of a function to be called by each subtask in the second model code, and the node identifier of each slave node; The processing module is further configured to generate a performance status table corresponding to the M slave nodes based on the multiple timestamp information and the multiple call function information, and determine the performance status of the M slave nodes according to the performance status table.

9. An electronic device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of the performance analysis method for a slave node as claimed in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the performance analysis method of the slave node according to any one of claims 1 to 7 are implemented.

Images