Model training method and device, electronic equipment and storage medium

Abstract

The invention provides a model training method and device, electronic equipment and a storage medium, and relates to the technical field of artificial intelligence system performance optimization and deep learning framework scheduling. The method comprises the following steps: in a current batch training process of a target model, acquiring execution information of a corresponding bottom kernel when a processor executes a plurality of computational operators of the target model; according to the execution information, determining a global key path, related to the total calculation duration, of the target model in the current batch training process, and identifying target operators forming training iteration delay in calculation operators contained in the global key path; and generating a delay optimization instruction for the target operator, and in response to the delay optimization instruction, adjusting training configuration parameters corresponding to the target operator in an upper-layer training framework of the processor so as to apply the adjusted training configuration parameters in a subsequent batch training process of the target model. According to the scheme, the training performance of model training can be improved.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence system performance optimization and deep learning framework scheduling technology, and more specifically, to a model training method, apparatus, electronic device and storage medium. Background Technology

[0002] With the rapid development of artificial intelligence technology, the scale and complexity of deep learning models continue to rise, and large models containing billions or even hundreds of billions of parameters have become the industry mainstream. The training process of such models is highly dependent on the efficient execution capabilities of the underlying hardware (such as image processors, tensor processors, etc.), and the training efficiency largely depends on the actual running performance of each operator in the computation graph on the hardware.

[0003] Current mainstream deep learning training frameworks (such as PyTorch, TensorFlow, and DeepSpeed) typically rely on pre-defined operator scheduling strategies and underlying execution configurations when performing model training. These configurations are mostly determined before training begins through static computation graph optimization, heuristic rules, or offline autotuning processes. Once training starts, it becomes difficult to dynamically adjust them based on the actual hardware behavior during runtime.

[0004] However, in real model training, different batches of input data, fluctuations in hardware resources, or differences in operator implementation may cause the actual execution delay of some computational operators to deviate significantly from expectations, thus becoming a bottleneck affecting the time consumption of the entire training iteration and resulting in low training performance.

[0005] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] The purpose of this disclosure is to provide a model training method, apparatus, electronic device, and storage medium that can improve the training performance of model training.

[0007] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part from practice of this disclosure.

[0008] According to a first aspect of the present disclosure, a model training method is provided, comprising: During the current batch training of the target model, the execution information of the underlying kernel corresponding to the processor executing multiple computational operators of the target model is obtained; Based on the execution information, determine the global critical path of the target model related to the total computation time during the current batch training process, and identify the target operator that constitutes the training iteration delay among the computation operators contained in the global critical path; Generate a latency optimization instruction for the target operator, and adjust the training configuration parameters corresponding to the target operator in the upper-layer training framework of the processor in response to the latency optimization instruction, so as to apply the adjusted training configuration parameters in subsequent batch training of the target model.

[0009] In some example embodiments of this disclosure, based on the foregoing scheme, determining the global critical path related to the total computation time of the target model during the current batch training process according to the execution information includes: The pre-built or trained operator performance model is updated based on the execution information to obtain the updated operator performance model; The updated operator performance model is used to determine the global critical path with the longest total computation time for the target model during the current batch training process.

[0010] In some exemplary embodiments of this disclosure, based on the foregoing scheme, updating the pre-built or trained operator performance model according to the execution information to obtain the updated operator performance model includes: The execution information is mapped to obtain operator performance, and operator-related features in the current batch training process are obtained; the operator-related features include at least one of operator input shape, scheduling parameters, and hardware features; A mapping relationship is constructed between the operator performance and the operator-related features, and the pre-constructed or trained operator performance model is updated based on the mapping relationship to obtain the updated operator performance model.

[0011] In some example embodiments of this disclosure, based on the foregoing scheme, determining the global critical path with the longest total computation time for the target model during the current batch training process using the updated operator performance model includes: The updated operator performance model is used to determine the computation time of each computation operator in the current batch training process of the target model. Based on the computation time of each of the aforementioned computation operators, the global critical path with the longest total computation time is determined.

[0012] In some example embodiments of this disclosure, based on the foregoing scheme, determining the global critical path with the longest total computation time based on the computation time of each of the computation operators includes: Obtain the training computation graph of the target model, wherein the training computation graph includes the sequential dependencies between the computation operators; Based on the sequential dependencies between the computation operators and the computation time of each computation operator, the global critical path with the longest total computation time is determined.

[0013] In some example embodiments of this disclosure, based on the foregoing scheme, determining the global critical path with the longest total computation time based on the sequential dependencies between the computation operators and the computation time of each computation operator includes: Based on the sequential dependencies between the computational operators, the global candidate path of the target model in the current batch training process is determined; Based on the computation time of each computation operator on the global candidate path, the total computation time of the global candidate path is determined, and the global candidate path with the longest total computation time is taken as the global critical path.

[0014] In some example embodiments of this disclosure, based on the foregoing scheme, the method is executed by a first device, and the method further includes: After the target model is trained, a target optimization instruction is generated based on each of the delay optimization instructions in the iterative training process; The computing power information of the first device and the target optimization instructions are sent to the second device; The computing power information of the first device is used to determine the computing power difference between the computing power information of the first device and the computing power information of the second device during the training of the target model by the second device. The computing power difference is used to adjust the training configuration parameters of the upper-layer training framework of the processor of the second device in combination with the target optimization instructions.

[0015] In some example embodiments of this disclosure, based on the foregoing scheme, the method further includes: After updating the pre-built or trained operator performance model according to the execution information, the updated operator performance model is obtained and sent to the second device; The updated operator performance model is used to iteratively train the target model during the training process of the target model by the second device.

[0016] In some example embodiments of this disclosure, based on the foregoing scheme, identifying the target operator constituting the training iteration delay among the computational operators included in the global critical path includes: Obtain the total computation time of the global critical path during the current batch training process, and obtain the historical computation time of the target model on the same global critical path during historical batch training processes; If the total computation time is greater than the historical computation time, then the target operator constituting the training iteration delay is identified among the computation operators included in the global critical path; the target operator in the current batch training process is different from the target operator in the historical batch training process.

[0017] In some example embodiments of this disclosure, based on the foregoing scheme, identifying the target operator constituting the training iteration delay among the computational operators included in the global critical path includes: For each computation operator contained in the global critical path, obtain the computation time of the computation operator and determine the dependency path to which the computation operator belongs; Based on the computation time of each computation operator and the dependency path to which it belongs, the target operator constituting the training iteration delay among the computation operators included in the global critical path is determined.

[0018] According to a second aspect of the present disclosure, a model training method is provided, the method being executed by a second device, the method comprising: The system receives computing power information and target optimization instructions from the first device after the target model training is completed; the target optimization instructions are generated based on the delay optimization instructions of the target model during the iterative training process, and the delay optimization instructions are obtained according to the above-mentioned model training method. Obtain the computing power information of the second device, and during the current batch training process of the target model deployed in the second device, determine the computing power difference between the computing power information of the second device and the computing power information of the first device; Based on the computing power difference and the target optimization instructions, the training configuration parameters of the upper-layer training framework of the processor of the second device are adjusted so that the adjusted training configuration parameters can be applied in subsequent batch training of the target model.

[0019] In some example embodiments of this disclosure, based on the foregoing scheme, adjusting the training configuration parameters of the processor upper-layer training framework of the second device based on the computing power difference and the target optimization instructions includes: The target optimization instruction is updated based on the computing power difference to obtain an updated target optimization instruction; the updated target optimization instruction is matched with the computing power information of the second device. In response to the updated target optimization instructions, the training configuration parameters of the processor upper-layer training framework of the second device are adjusted.

[0020] In some example embodiments of this disclosure, based on the foregoing scheme, the method further includes: The system receives an updated operator performance model sent by the first device. The updated operator performance model is obtained by the first device updating a pre-built or trained operator performance model based on the execution information of the processor's underlying kernel during the training of the target model. The updated operator performance model is loaded, and the target model deployed in the second device is iteratively trained based on the updated operator performance model.

[0021] According to a third aspect of the present disclosure, a model training apparatus is provided, comprising: The execution information acquisition module is used to acquire the execution information of the underlying kernel corresponding to the processor executing multiple computational operators of the target model during the current batch training process of the target model. The training iteration delay analysis module is used to determine the global critical path of the target model related to the total computation time during the current batch training process based on the execution information, and to identify the target operators that constitute the training iteration delay among the computation operators contained in the global critical path. The first training configuration adjustment module is used to generate a latency optimization instruction for the target operator, and adjust the training configuration parameters corresponding to the target operator in the upper-layer training framework of the processor in response to the latency optimization instruction, so as to apply the adjusted training configuration parameters in subsequent batch training of the target model.

[0022] According to a fourth aspect of the present disclosure, a model training apparatus is provided, executed by a second device, the apparatus comprising: A receiving module is used to receive the computing power information and target optimization instructions sent by the first device after the target model training is completed; the target optimization instructions are generated based on the delay optimization instructions of the target model during the iterative training process, and the delay optimization instructions are obtained according to the above-mentioned model training method; The computing power difference calculation module is used to obtain the computing power information of the second device and determine the computing power difference between the computing power information of the second device and the computing power information of the first device during the current batch training process of the target model deployed in the second device. The second training configuration adjustment module is used to adjust the training configuration parameters of the upper-layer training framework of the processor of the second device based on the computing power difference and the target optimization instructions, so as to apply the adjusted training configuration parameters in the subsequent batch training process of the target model.

[0023] According to a fifth aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory storing computer-readable instructions that, when executed by the processor, implement a model training method as described in the first or second aspect.

[0024] According to a sixth aspect of the present disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the model training method as described in the first or second aspect.

[0025] The technical solutions provided in this disclosure can include the following beneficial effects: The model training method in the example embodiments of this disclosure acquires the execution information of the underlying kernels corresponding to multiple computational operators of the target model during the current batch training process. This execution information reflects the actual performance data of the target model during runtime (rather than static prediction). Based on this execution information, the global critical path related to the total computation time of the target model during the current batch training process can be determined, and the target operators constituting training iteration delays in the global critical path can be identified. This achieves accurate positioning of the actual performance bottlenecks during the target model training process. Consequently, delay optimization instructions for the target operators can be generated, and the training configuration parameters corresponding to the target operators in the upper-layer training framework of the processor can be adjusted in response to the delay optimization instructions. The adjusted training configuration parameters can then be applied in subsequent batch training processes of the target model to improve the training performance of the target model in subsequent batch training processes. In other words, based on the execution information of the underlying kernels during the target model training process, delay optimization instructions that solve the performance bottlenecks during model training can be fed back to the upper-layer training framework to adaptively and dynamically adjust the training configuration parameters of the target operators. This allows the adjusted target operators to achieve model training faster, thereby improving the training performance of the model.

[0026] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0027] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0028] Figure 1 The schematic diagram illustrates a flowchart of a model training method according to some embodiments of the present disclosure.

[0029] Figure 2 The illustration shows a flowchart of determining the global critical path based on execution information according to some embodiments of the present disclosure.

[0030] Figure 3 The illustration shows a schematic diagram of the process of migrating computing power information and target optimization instructions from a first device to a second device according to some embodiments of the present disclosure.

[0031] Figure 4 The schematic diagram illustrates a flowchart of a model training method according to other embodiments of the present disclosure.

[0032] Figure 5 A schematic diagram of a model training apparatus according to some embodiments of the present disclosure is shown.

[0033] Figure 6 A schematic diagram of a model training apparatus according to other embodiments of the present disclosure is shown.

[0034] Figure 7 The schematic diagram illustrates the structural schematic of a computer system of an electronic device according to some embodiments of the present disclosure.

[0035] Figure 8 A schematic diagram of a computer-readable storage medium according to some embodiments of the present disclosure is shown.

[0036] In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts. Detailed Implementation

[0037] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this specification as detailed in the appended claims.

[0038] Furthermore, the accompanying drawings are for illustrative purposes only and are not necessarily drawn to scale. The block diagrams shown in the drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0039] In this example embodiment, a model training method is first provided. This model training method can be applied to electronic devices. For example, the electronic devices can be various computing devices with deep learning training capabilities and underlying hardware performance awareness capabilities, including but not limited to artificial intelligence training servers, master control nodes of distributed training clusters, scheduling controllers of cloud-native AI (Artificial Intelligence) platforms, edge intelligent training devices, dedicated training hosts with heterogeneous computing resources of GPU (Graphics Processing Unit) / CPU (Central Processing Unit) and kernel-level performance monitoring interfaces, and intelligent operation and maintenance servers that integrate operator execution tracing modules and dynamic configuration adjustment functions.

[0040] Figure 1 The schematic diagram illustrates a flowchart of a model training method according to some embodiments of the present disclosure. Reference Figure 1 As shown, the model training method may include the following steps: Step S110: During the current batch training process of the target model, obtain the execution information of the underlying kernel corresponding to the processor executing multiple computational operators of the target model; Step S120: Based on the execution information, determine the global critical path of the target model related to the total computation time in the current batch training process, and identify the target operators that constitute the training iteration delay among the computation operators contained in the global critical path. Step S130: Generate a latency optimization instruction for the target operator, and adjust the training configuration parameters corresponding to the target operator in the upper-layer training framework of the processor in response to the latency optimization instruction, so as to apply the adjusted training configuration parameters in subsequent batch training of the target model.

[0041] According to the model training method in this example embodiment, during the current batch training of the target model, the execution information of the underlying kernel corresponding to the processor executing multiple computational operators of the target model is obtained. This execution information reflects the actual performance data of the target model during runtime (rather than static prediction). Based on this execution information, the global critical path related to the total computation time of the target model during the current batch training can be determined, and the target operators constituting the training iteration delay among the computational operators included in the global critical path can be identified. This achieves accurate positioning of the actual performance bottleneck during the target model training process, thereby generating delay optimization instructions for the target operators. In response to the delay optimization instructions, the training configuration parameters corresponding to the target operators in the upper-layer training framework of the processor are adjusted so that the adjusted training configuration parameters can be applied in subsequent batch training processes of the target model, improving the training performance of the target model in subsequent batch training processes. In other words, based on the execution information of the underlying kernel during the target model training process, delay optimization instructions that solve the performance bottleneck during model training can be fed back to the upper-layer training framework to adaptively and dynamically adjust the training configuration parameters of the target operators, so that the adjusted target operators can achieve model training faster, thereby improving the training performance of the model training.

[0042] The model training method in this example embodiment will be further explained below.

[0043] In step S110, during the current batch training process of the target model, the execution information of the underlying kernel corresponding to the processor executing multiple computation operators of the target model is obtained.

[0044] The target model refers to the deep learning model to be trained, whose structure can include, but is not limited to, Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), Transformers (a neural network architecture based on self-attention mechanisms), Multilayer Perceptrons (MLPs), or combinations thereof. During training, the target model can be represented as a Directed Acyclic Graph (DAG) composed of multiple computational operators. Each operator corresponds to a specific mathematical operation, such as matrix multiplication, convolution, activation functions (e.g., ReLU, GELU), normalization (e.g., LayerNorm, BatchNorm), or communication operations (e.g., AllReduce, Broadcast).

[0045] The current batch training process refers to the process by which the target model processes a batch of input data in one forward and backward propagation iteration. This process can be completed within a single training loop and may be repeated thousands to millions of times to complete the entire training task.

[0046] A processor is a hardware processing unit used to perform computational tasks, which may include, but is not limited to, a central processing unit, a graphics processing unit (CPU), a tensor processing unit (TPU), a neural processing unit (NPU), or other dedicated accelerators with parallel computing capabilities. In some deployment scenarios, multiple computational operators can be scheduled to be executed on the processor by a deep learning training framework, which is mapped to calls to one or more underlying kernels.

[0047] A kernel is the smallest schedulable program unit that actually performs computational tasks on a processor. For example, in a GPU environment, a kernel can be a CUDA (Compute Unified Device Architecture) kernel or a HIP (Heterogeneous-computing Interface for Portability) kernel; in a TPU environment, a kernel can be a low-level computational fragment compiled from XLA (Accelerated Linear Algebra). Each computational operator can correspond to one or more kernels at runtime, depending on operator fusion, partitioning strategies, and hardware characteristics.

[0048] Execution information refers to performance data that reflects the actual running status of the underlying kernel on the processor. Execution information may include, but is not limited to: kernel start timestamp, end timestamp, actual execution duration, SM (Streaming Multiprocessor) utilization, memory bandwidth utilization, memory access volume, compute utilization, number of streaming multiprocessors used, size of shared memory used, number of registers used, instruction throughput, cache hit rate, and the identifier of the CUDA stream or execution queue to which it belongs.

[0049] In some embodiments, this execution information can be collected in real time through hardware performance counters, driver layer interfaces (such as the Compute Unified Device Architecture Profiling Tools Interface), or operating system-level tracing tools (such as Linuxperf, eBPF).

[0050] In some embodiments, execution information can be obtained without interrupting the training process. That is, while the current batch training is running normally, the execution information of each underlying kernel corresponding to the processor executing multiple computational operators of the target model can be recorded synchronously through a non-intrusive or lightweight instrumentation mechanism.

[0051] For example, a performance monitoring callback function can be embedded in the scheduler unit of a deep learning training framework. While the current batch of training is executing normally, when the underlying kernel submits to the device queue or completes execution, the performance monitoring callback function can automatically trigger the reporting of execution information.

[0052] In other embodiments, the collection of execution information can be limited to a set of key operators, rather than all operators, to reduce monitoring overhead. For example, high computational density operators (such as linear layers in an MLP) can be pre-marked as monitoring objects in each computational operator, and only the execution information of the underlying kernel corresponding to the high computational density operators can be collected.

[0053] Understandably, by obtaining the execution information of the underlying kernel during the current batch training process, we can truly reflect the operating efficiency of each computational operator in the actual hardware environment, providing a reliable data foundation for the subsequent dynamic identification of performance bottlenecks and avoiding the bias caused by relying on static analysis or offline prediction.

[0054] In step S120, based on the execution information, the global critical path related to the total computation time of the target model in the current batch training process is determined, and the target operator constituting the training iteration delay is identified among the computation operators contained in the global critical path.

[0055] The global critical path refers to one or more paths related to the total computation time during the current batch training of the target model. Each global critical path contains computational operators that are sequentially connected according to data or control dependencies, extending from the starting node of the training iteration (e.g., data input or parameter loading) to the ending node (e.g., optimizer update completion). The computational operators contained in the global critical path can be some or all of the multiple computational operators of the target model.

[0056] For example, the global critical path can be the path with the longest total computation time, or it can be one or more paths with a total computation time exceeding a specified time threshold.

[0057] In some embodiments, the computation time of each computation operator is extracted based on the execution information, and combined with the dependencies in the training computation graph of the target model, the path time is accumulated layer by layer from the input end to the output end using topological sorting and dynamic programming methods to determine the total computation time of each path in the training computation graph, and the path related to the total computation time (such as the one with the longest total computation time) is determined as the global critical path.

[0058] For example, during the training of a target model based on the Transformer architecture, the following data can be collected: 0.8 milliseconds for the Embedding operator during the forward propagation phase, 2.5 milliseconds for the QKV (Query, Key, Value) projection operator, 3.2 milliseconds for the self-attention computation operator, and 1.9 milliseconds for the linear layer operator of the feedforward network. During the backpropagation phase, the gradient synchronization (AllReduce) communication operator takes 4.0 milliseconds. Based on the computational graph dependency analysis, the QKV projection operator, the self-attention computation operator, and the gradient synchronization communication operator form a sequential dependency chain with a cumulative duration of 9.7 milliseconds, exceeding the duration of other parallel or non-critical branch paths. Therefore, this sequential dependency chain is considered the global critical path in the current batch training process.

[0059] In other embodiments, a pre-built or trained operator performance model can be used to determine the global critical path of the target model that is related to the total computation time (e.g., the longest total computation time) during the current batch training process.

[0060] For example, execution information can be input into a pre-trained operator performance model, which outputs the expected execution time of each computational operator under the current hardware environment, and derives the critical path by combining the computation graph structure. This operator performance model can be a regression model, a graph neural network (GNN), or a lightweight decision tree model trained based on historical execution data, used to establish the mapping relationship between operator features (such as input tensor shape, data type, and hardware model) and their execution time.

[0061] In other embodiments, other methods can be used to determine the global critical path, such as extracting the longest subsequence composed of the user-specified set of critical operators (such as all Attention or Linear layers), or combining the hardware utilization curve (such as the time period when GPU SM activity is consistently high) to infer the corresponding operator interval as a critical path candidate, etc., and are not limited to these.

[0062] The target operator is a computational operator that, among the computational operators contained in the global critical path, makes a significant contribution to the latency of the current training iteration and whose execution configuration has adjustable space.

[0063] In some embodiments, identifying the target operator constituting the training iteration delay among the computation operators included in the global critical path includes: for each computation operator included in the global critical path, obtaining the computation time of the computation operator and determining the dependency path to which the computation operator belongs; and determining the target operator constituting the training iteration delay among the computation operators included in the global critical path based on the computation time of each computation operator and the dependency path to which it belongs.

[0064] In this context, a dependency path refers to a directed path taken by a computation operator (or node) from each of the preceding operators that influence its execution to reach the computation operator. These preceding operators can be the original input nodes or any intermediate operators that generate the input tensors required for the computation operator. Therefore, a dependency path can be a global path that runs through the entire computation process or a local path that starts from a certain intermediate node.

[0065] For example, if operator B depends on the output of operator A, and operator A itself depends on another operator C, then for operator B, its dependency path includes at least the connection between A and C, and the complete path from C to A and then to B.

[0066] Specifically, we can determine the computation operators on the dependency path to which the computation operator belongs, sum up the computation times of each computation operator on the dependency path to obtain the computation time of the dependency path to which each computation operator belongs, determine the dependency path with the longest computation time based on the computation time of the dependency path to which each computation operator belongs, and take the computation operator on the dependency path with the longest computation time as the target operator.

[0067] In other embodiments, the computation time of each computation operator can be obtained, and computation operators whose computation time exceeds a preset time threshold or whose delay contribution (i.e., the proportion of the execution time of the computation operator to the total time of the critical path) exceeds a preset delay sharing degree can be selected as target operators. The preset time threshold can be set as needed.

[0068] For example, on the global critical path, multiple computation operators A and B are executed in parallel, and the total computation time of multiple computation operators A is greater than the computation time of computation operator B. In this case, multiple computation operators A can all be used as target operators.

[0069] For example, on the global critical path, there are multiple serial computation operators that can be parallelized. If the computation time of the multiple parallel computation operators (i.e., the time of the computation operator with the longest computation time) is less than the total computation time of the serial computation, then all of the multiple serial computation operators can be used as target operators.

[0070] In other embodiments, the computation time of each computation operator can also be obtained, and the one or more computation operators with the longest computation time in the global critical path can be used as the target operator, or the computation operator that appears in the critical path in multiple consecutive batches of training and ranks high in computation time can be used as the target operator, or the operator that can significantly shorten the total computation time of the global critical path after removal can be used as the target operator, etc., without limitation.

[0071] For example, in the training process of a target model based on the Transformer architecture, if the self-attention computation operator takes 3.2 milliseconds in the global critical path, accounting for 35% of the total time of the critical path, and this computation operator can be accelerated by enabling a high-performance attention computation kernel or adjusting the shared memory usage strategy, then this self-attention computation operator can be identified as the target operator.

[0072] For example, in data-parallel training, if the gradient synchronization operator becomes a high-latency node at the end of the critical path due to the large amount of communication, and it can be optimized by adjusting the bucket size or the communication flow priority, then the gradient synchronization operator can also be used as the target operator.

[0073] In addition, weighted filtering can be performed by combining the optimizability of the operator (such as whether it supports fusion or whether there are multiple kernel implementations), and the computational operators with configuration adjustment space can be selected as the target operators.

[0074] Understandably, by determining the global critical path based on real execution information and accurately identifying the target operators within it, optimization resources can be focused on the bottlenecks that truly affect end-to-end training performance, avoiding ineffective adjustments to non-critical path operators, thereby improving the effectiveness of subsequent optimization instructions and the overall training efficiency of the system.

[0075] In some implementations, if there are multiple candidate critical paths with similar total computation time during the current batch training process (e.g., the difference is less than 5%), the multiple candidate critical paths can be merged into a critical path set, and high-frequency, high-time-consuming operators that co-occur across paths can be comprehensively identified as target operators to enhance the robustness of the identification results.

[0076] In other implementations, historical iteration data can be introduced to assist in identifying target operators. For example, the global critical path in the current batch training process can be compared with the average critical path of several historical batches. If a certain computational operator becomes the latency bottleneck of the critical path for the first time in the current batch training process, it may reflect a sudden change in hardware state or an abnormal input. At this time, a more granular diagnostic or conservative optimization strategy can be triggered.

[0077] In step S130, a delay optimization instruction for the target operator is generated, and in response to the delay optimization instruction, the training configuration parameters corresponding to the target operator in the upper-layer training framework of the processor are adjusted so as to apply the adjusted training configuration parameters in subsequent batch training of the target model.

[0078] Delay optimization instructions are control information used to instruct the upper-level training framework of the processor to perform configuration adjustments on the target operator. These instructions can be generated by the analysis module or optimization controller during training, and their content is matched to the type of the target operator, its current performance, and the hardware environment.

[0079] Delay optimization instructions can include, but are not limited to, operator scheduling policies, operator fusion, communication packet adjustment, pipeline parallelism depth adjustment, kernel launch parameter adjustment, scheduling priority or distribution policies, etc. The specific form of delay optimization instructions can be a structured configuration message, a set of key-value pairs of parameters, a policy descriptor, or the parameter payload of a calling interface, etc.

[0080] The upper-level training framework refers to the software system used to schedule and execute deep learning model training tasks, including but not limited to heterogeneous acceleration frameworks such as PyTorch, Megatron, DeepSpeed, CUDA, and MUSA (Matrix Unified Software Architecture) or their extended versions. This training framework is responsible for managing computation graph construction, operator scheduling, memory allocation, automatic differentiation, and low-level kernel calls, and provides configurable interfaces to support runtime parameter tuning.

[0081] Training configuration parameters are adjustable parameters that affect the execution behavior of the target operator on the processor. Adjusting them does not change the logical function of the model, but can significantly affect the execution efficiency.

[0082] In some embodiments, training configuration parameters may include underlying kernel startup parameters corresponding to the operators, such as the block dimension (blockDim), grid dimension (gridDim), shared memory size, or stream allocation identifier of the CUDA kernel. In other embodiments, training configuration parameters may also include operator fusion switches, communication grouping strategies (such as the bucket size of gradient synchronization communication operators), tensor layout formats (such as NCHW vs NHWC), precision modes (such as FP16 / TF32 / INT8), parallel partitioning methods, or cache prefetching strategies.

[0083] In some embodiments, delay optimization instructions for the target operator can be matched from a predefined optimization strategy library based on the type of the target operator and its associated execution information.

[0084] For example, if the target operator is a convolution operator and the current use of the default cuDNN (CUDA Deep Neural Network library) algorithm results in low SM utilization, the delay optimization instruction can specify switching to the more efficient Winograd algorithm or enabling Tensor Core acceleration; if the target operator is a gradient synchronization communication operator and bandwidth is not observed to be saturated, the delay optimization instruction can suggest increasing the bucket size to improve throughput.

[0085] In other embodiments, a lightweight search mechanism can be used to dynamically generate delay optimization instructions for the target operator.

[0086] For example, given the input shape and hardware model of the target operator, a rapid trial run is performed in a limited candidate set (such as 3–5 kernel implementations), and the configuration with the lowest measured latency is selected as the latency optimization instruction, and the results are cached for subsequent reuse.

[0087] In other implementations, latency optimization instructions can also include multi-parameter combination strategies, such as simultaneously adjusting operator fusion granularity and communication overlap depth; or dynamically generating instructions using reinforcement learning agents based on historical feedback; or encapsulating optimization instructions into portable configuration snapshots for cross-device migration. All of the above methods can be flexibly selected according to actual deployment needs, and are not limited here.

[0088] It is understandable that after generating the delay optimization instruction for the target operator, the delay optimization instruction can be sent to the upper-level training framework. The upper-level training framework can adjust the training configuration parameters corresponding to the target operator in response to the delay optimization instruction, and obtain the adjusted target operator. Then, the target model applies the adjusted training configuration parameters in subsequent batch training processes. When the processor executes the adjusted target operator, the computation time is reduced, as is the total computation time of the global critical path, thereby improving the overall training throughput and hardware resource utilization, and improving the training performance of the model.

[0089] Furthermore, it significantly enhances the performance adaptability of the training process. By collecting execution information from the underlying kernel in real time, it dynamically detects computational bottlenecks and automatically adjusts optimization strategies, effectively addressing runtime disturbances such as changes in data distribution and fluctuations in hardware status. Simultaneously, by identifying global critical paths related to total computation time and accurately locating the target operators constituting training iteration delays, abstract performance problems can be transformed into concrete, interpretable computational nodes, enabling visualized analysis of performance bottlenecks. Ultimately, under the same hardware conditions, by dynamically optimizing the training configuration parameters of target operators, overall training efficiency can be improved by 10% to 30%, significantly accelerating model iteration and delivery.

[0090] The contents of steps S110 to S130 will be described in detail below.

[0091] In an example embodiment of this disclosure, such as Figure 2 As shown, the following steps can be used to determine the global critical path related to the total computation time of the target model during the current batch training process based on the execution information. Specifically, these steps may include S210 to S220: Step S210: Update the pre-built or trained operator performance model based on the execution information to obtain the updated operator performance model.

[0092] The operator performance model refers to a machine learning model or parameterized function used to predict or estimate the execution latency of various computational operators under specific hardware conditions. The input of the target model may include operator type, input tensor shape (such as batch size, number of channels, sequence length), data type (such as FP16, FP32), hardware characteristics (such as GPU model, memory bandwidth, number of SMs), and historical execution statistics (such as average latency, variance), and the output is the expected execution time or relative performance score of the computational operator under the current conditions.

[0093] In some embodiments, an operator performance model can be constructed using an event-driven performance model simulation method and embedded into the software stack, so that the operator performance model can be called subsequently.

[0094] In other embodiments, the operator performance model may be a lightweight regression-based model (such as a decision tree, a linear model), a graph neural network, or an experience lookup table trained from offline automatically tuned data, etc., and is not limited thereto.

[0095] In some embodiments, the operator performance model can be pre-built. It can be an empirical model or an analytical model built based on hardware specifications and operator parameters. Alternatively, it can be pre-built using techniques such as offline micro-benchmarking, compiler scheduling information, or historical training task logs, and integrated into the training framework in a callable form. No limitation is imposed here.

[0096] In other embodiments, the operator performance model can also be pre-built and pre-trained. In some implementations, pre-training can be performed during system initialization using performance analysis data from historical training tasks to obtain a pre-trained operator performance model, which is then deployed in the optimization controller of the training framework. In other implementations, the operator performance model can be trained simultaneously during the pre-batch training of the target model to obtain a pre-trained operator performance model. In other implementations, the pre-trained operator performance model can be obtained through supervised training based on offline automatic tuning data, or through training based on compiler intermediate representations and static modeling of the hardware analyzer, etc., without limitation.

[0097] In some embodiments, the model parameters of a pre-built or trained operator performance model can be adjusted based on execution information to obtain an updated operator performance model.

[0098] Here, model parameters refer to the learnable or configurable values ​​within the operator performance model used to establish the mapping relationship between input features and prediction latency. In machine learning-based implementations, model parameters may include weights and bias parameters in neural networks, split thresholds and leaf node values ​​in decision trees, or coefficient vectors in regression models; in empirical formula-based implementations, model parameters may also include hardware-related calibration factors (such as memory bandwidth utilization coefficients, computational unit efficiency factors) or baseline latency constants implemented in the kernel.

[0099] Specifically, the actual computation time of the corresponding computation operator is obtained based on the execution information, and the predicted computation time of the computation operator output by the pre-built or trained operator performance model is obtained; based on the difference between the actual computation time and the predicted computation time, the model parameters of the operator performance model are adjusted (online fine-tuning or incremental update) to obtain the updated operator performance model.

[0100] For example, the latency baseline value of a certain type of convolution operator can be updated using the Exponential Moving Average (EMA) method, or the weights of the neural network performance model can be adjusted by a small gradient step size, thereby obtaining an updated operator performance model that better fits the current hardware status and load environment.

[0101] In other embodiments, updating a pre-built or trained operator performance model based on execution information to obtain an updated operator performance model includes: mapping the execution information to obtain operator performance, and obtaining operator-related features in the current batch training process; the operator-related features include at least one of operator input shape, scheduling parameters, and hardware features; constructing a mapping relationship between operator performance and operator-related features, and updating the pre-built or trained operator performance model based on the mapping relationship to obtain an updated operator performance model.

[0102] Operator performance refers to a quantitative metric used to characterize the actual efficiency of a computation operator in a specific execution context. Operator performance can be characterized by execution latency or computation time (in milliseconds or microseconds), or by dimensions such as throughput, resource utilization, or overall performance. Operator performance reflects the end-to-end behavior of the operator in a real hardware environment and is a key input for subsequent performance modeling and optimization decisions.

[0103] Operator-related features describe key contextual information affecting operator execution efficiency during the current batch training process. Specifically, operator input shape refers to the dimensionality of the tensors processed by the computation operator, such as batch size, sequence length, number of channels, and hidden dimensions; scheduling parameters refer to the execution configuration allocated to the computation operator by the training framework or compiler, such as the grid dimension, block dimension, shared memory size, stream identifier used, and whether operator fusion is enabled in the CUDA kernel; hardware features refer to the physical or logical attributes of the execution environment, such as processor model (e.g., A100, H100), memory capacity, driver version, CUDA version, PCIe (Peripheral Component Interconnect Express) bandwidth, and whether Tensor Cores are enabled. Operator-related features can be recorded by the training framework during the operator scheduling phase and stored in association with the corresponding kernel execution information.

[0104] In some embodiments, the underlying kernels belonging to the same computational operator can be identified based on the pre-built affiliation relationship between the underlying kernels and computational operators; the execution information of the underlying kernels belonging to the same computational operator is aggregated to obtain the operator performance of the computational operator in the current batch training.

[0105] The attribution relationship between the underlying kernel and the computation operator can be established in several ways. In one implementation, the upper-layer training framework attaches a unique operator identifier (such as operator name, graph node ID, or hash value) to each generated underlying kernel when scheduling computation operators, and synchronously records this identifier when collecting execution information through a performance analysis interface (such as Nsight or eBPF), thereby directly associating the underlying kernel with the computation operator and constructing the attribution relationship between the underlying kernel and the computation operator. In another implementation, the higher-level computation operator to which the underlying kernel belongs can be inferred based on the naming rules of kernel function names (such as including semantic keywords such as "flash_attn", "gemm", "layernorm") or the temporal continuity of lower-level kernel calls (such as the interval between adjacent kernels being less than a preset threshold and no other operators interspersed).

[0106] Specifically, multiple kernel test data belonging to the same operator can be aggregated to generate operator performance. If the underlying kernel corresponding to the computation operator is executed serially (such as the continuous computation phase in forward propagation), the execution time of each serial underlying kernel is summed as the computation time of the computation operator; if the kernels are executed in parallel (such as independent computation by each head in multi-head attention or parallel computation by multiple experts), the difference between the latest end time and the earliest start time of each parallel underlying kernel is taken, or the maximum single kernel time is directly taken as the operator performance; in other cases, the execution time can also be weighted and corrected by combining hardware utilization indicators (such as SM activity and memory bandwidth saturation) to form a comprehensive performance score, which is not limited here.

[0107] In other embodiments, other methods may be used to map execution information to obtain operator performance, which are not limited here.

[0108] In some embodiments, after mapping the execution information of the underlying kernel to obtain the operator performance, the current operator-related features can be read directly from the runtime interface of the upper training framework, or the current operator-related features can be parsed from the compiler intermediate representation or scheduling log, or the hardware context can be obtained from system monitoring tools, etc., and are not limited to these.

[0109] A mapping function can be used to construct a mapping relationship between operator performance and operator-related features. This mapping relationship can then be used as samples to update the pre-built or trained operator performance model, resulting in an updated operator performance model. It is understandable that different computational operators have different operator-related features in the current batch training process, such as operator input shape, scheduling parameters, and hardware characteristics. Therefore, the execution information generated by the processor when executing each computational operator of the target model is also different, and each piece of execution information can be mapped to the corresponding operator's performance.

[0110] Specifically, multiple operator performances and their corresponding operator-related features are input into a preset mapping function. The preset mapping function is used to fit the multiple operator performances and their corresponding operator-related features to obtain mapping parameters. These mapping parameters are then filled into the preset mapping function to obtain an updated mapping function. This updated mapping function represents the mapping relationship between operator performances and operator-related features.

[0111] In a regression-based implementation, the sample can be added to an online learning buffer, and the model parameters can be adjusted using gradient descent or exponential moving average. In a lookup table-based implementation, the new observation can be used to directly replace or weight the existing entries.

[0112] Understandably, the updated operator performance model can more accurately reflect the actual execution behavior of each operator under the current software and hardware environment, thus providing a more reliable predictive basis for determining the global critical path.

[0113] Step S220: Using the updated operator performance model, determine the global critical path with the longest total computation time for the target model during the current batch training process.

[0114] In some embodiments, the identifiers of each computational operator in the current batch training process of the target model can be input into the updated operator performance model. The updated operator performance model can determine the computation time of each computational operator and the global critical path with the longest total computation time in the current batch training process of the target model.

[0115] In other embodiments, the global critical path with the longest total computation time for the target model during the current batch training is determined by using the updated operator performance model, including: determining the computation time of each computation operator for the target model during the current batch training by using the updated operator performance model; and determining the global critical path with the longest total computation time based on the computation time of each computation operator.

[0116] Understandably, the operator-related features (including operator input shape, scheduling parameters, and hardware features) of each computational operator during the current batch training process of the target model can be input into the updated operator performance model. The updated operator performance model then outputs the predicted execution time of the corresponding computational operator, which serves as its computation time in the current batch. Since this operator performance model has undergone dynamic calibration based on real underlying kernel execution information in the preceding steps, its prediction results can more accurately reflect the actual latency performance of each operator under the current software and hardware environment.

[0117] Specifically, based on the computation time of each computation operator, the global critical path with the longest total computation time is determined, including: obtaining the training computation graph of the target model, which includes the sequential dependencies between each computation operator; and determining the global critical path with the longest total computation time based on the sequential dependencies between each computation operator and the computation time of each computation operator.

[0118] The training computation graph refers to a directed acyclic graph (DAG) consisting of all computational operators and their data dependencies involved in the target model during the current training iteration. Each node in the training computation graph represents a computational operator, and each directed edge indicates that the output of the source operator is the input of the target operator, or that there is a control dependency (such as gradient dependency or communication synchronization point in backpropagation). This training computation graph can be generated by the automatic differentiation engine or graph optimizer during forward or backward propagation in the training framework and stored in memory in an intermediate representation for access by the analysis module.

[0119] Sequence dependency refers to a logical or data constraint in a training computation graph where a computation operator can only begin execution after one or more of its predecessor operators have completed. This sequence dependency determines the execution order between operators and is the foundation for constructing a directed acyclic graph.

[0120] Sequence dependencies can be categorized into data dependencies and control dependencies. Data dependencies refer to situations where the input tensor of the current operator is provided by the output tensor of the predecessor operator. For example, when a ReLU (Rectified Linear Unit) activation function is applied after a linear layer, ReLU must wait for the linear unit to complete its computation and generate its output before execution; a clear data dependency exists between the two. Control dependencies refer to situations where, even without direct data transfer, certain operations can only be triggered after a specific operator has completed its computation. For example, gradient computation in backpropagation must wait for all forward computations to complete; similarly, gradient synchronization communication operations must wait for all gradients involved in the communication to complete their computations before initiating. These dependencies are explicitly inserted by automatic differentiation engines or distributed training frameworks.

[0121] Specifically, based on the sequential dependencies between computation operators and the computation time of each computation operator, the global critical path with the longest total computation time is determined, including: determining the global candidate paths of the target model in the current batch training process according to the sequential dependencies between computation operators; determining the total computation time of the global candidate paths based on the computation time of each computation operator on the global candidate paths, and taking the global candidate path with the longest total computation time as the global critical path.

[0122] In this context, a global candidate path refers to a feasible execution path that satisfies a sequential dependency relationship between input nodes (such as data loading operators, placeholders, or model inputs) and output nodes (such as loss functions, optimizer update operations, or gradient application nodes) in the training computation graph. Each global candidate path consists of a series of computation operators connected in a dependency order, and there are no loops within the path, conforming to the topological characteristics of a directed acyclic graph.

[0123] In the specific implementation, the training computation graph can be traversed, and all feasible global candidate paths from input nodes (such as data loading or placeholders) to output nodes (such as loss functions or optimizer updates) can be extracted according to the sequential dependencies between each computation operator. The computation time of the computation operator on each global candidate path is accumulated to obtain the total computation time of the global candidate path. The path with the largest total computation time is determined as the global critical path.

[0124] In other embodiments, dynamic programming can be used to avoid full path enumeration. That is, the earliest completion time is calculated node by node in topological order, and the longest path is obtained by backtracking at the output node, thereby reducing computational overhead while ensuring accuracy. The specific implementation method can be flexibly selected according to the size of the computation graph and the real-time requirements, and is not limited here.

[0125] Understandably, by introducing an online-updable operator performance model, the performance distribution of the entire training computation graph can still be reasonably inferred even in the absence of complete real-time tracking or when some operators are not sampled, thus improving the robustness and generalization ability of global critical path identification. Simultaneously, by leveraging the online construction and continuous updating of the operator performance model, combined with the closed-loop feedback mechanism of delayed optimization instructions, the traditional tuning cycle relying on manual trial and error is significantly shortened, enabling the system to converge to an efficient configuration within a few iterations. Furthermore, optimization experience validated on the first device is encapsulated in the form of target optimization instructions and updated operator performance models. This allows for automatic adaptation on the second device based on local computing power information, eliminating the need for manual retuning and enabling the reuse of historical strategies in heterogeneous hardware environments, significantly reducing deployment costs.

[0126] In some embodiments, the process of performance feedback optimization based on a processor-level training framework (e.g., PyTorch) includes: injecting an execution information collection hook into the autograd graph executor of the target model; recording the execution information of the underlying kernel through the hook during each training iteration, which may include execution time, data throughput, and underlying kernel type; updating the pre-built operator performance model online based on the collected execution information; triggering a delay optimization instruction generation mechanism when an increase in the total computation time of the global critical path is detected, adjusting the training configuration parameters of the corresponding target operator (e.g., operator parallelism, kernel selection strategy, or communication packet size); and after the target model training converges, aggregating the delay optimization instructions generated and verified during the iteration process to form target optimization instructions that match the current hardware environment, and persisting them as a hardware configuration file for subsequent deployment or cross-device migration.

[0127] In one example embodiment of this disclosure, the action is performed by a first device, such as... Figure 3 As shown, the above method may further include the following steps S310 to S320: Step S310: After the target model training is completed, generate the target optimization instruction based on the delay optimization instructions in the iterative training process.

[0128] Among them, the target optimization instruction refers to the comprehensive optimization configuration information obtained by aggregating, filtering or fusing multiple delay optimization instructions generated during the training process after the target model training is completed. It is used to instruct the second device to reuse the verified and effective performance tuning strategy in subsequent training tasks.

[0129] Specifically, after the target model is trained, the various delay optimization instructions generated during the iterative training process are aggregated to generate a comprehensive target optimization instruction. This aggregation process may include, but is not limited to, removing duplicate or conflicting instructions, retaining verified and effective optimization terms, categorizing and merging them by operator type or functional module, and encapsulating them in a structured form as a transferable configuration template.

[0130] For example, if the attention operator is switched to a high-performance attention computation kernel during multiple batch training processes and the latency is significantly reduced, then this strategy is solidified in the target optimization instruction; if the size of the gradient synchronization communication bucket stabilizes at 256 MB after multiple adjustments, then it is included as a recommended value in the target optimization instruction.

[0131] It is understandable that the target optimization instructions can take the form of configuration files, policy descriptors, optimization template identifiers, or parameter patch packages, etc., to reuse verified optimization experience on the same or different devices in the future, avoid repeated tuning, and improve training and deployment efficiency.

[0132] Step S320: The computing power information of the first device and the target optimization instructions are sent to the second device; wherein, the computing power information of the first device is used to determine the computing power difference between the computing power information of the first device and the computing power information of the second device during the training of the target model on the second device, and the computing power difference is used to adjust the training configuration parameters of the upper-layer training framework of the processor of the second device in combination with the target optimization instructions.

[0133] Among them, computing power information is used to characterize the computing power and resource characteristics of the device, which may include, but is not limited to: processor type (such as GPU / TPU / NPU model), video memory / memory capacity, theoretical floating-point computing power, memory bandwidth, interconnect topology (such as NVLink support), driver and software stack version (such as CUDA 12.1, PyTorch 2.3), etc.

[0134] The second device refers to another computing node that subsequently participates in the training of the target model. The hardware configuration of the second device can be the same as that of the first device, or they can differ. For example, the first and second devices may have different GPU models (e.g., the first device is model A and the second device is model B), different memory capacities, different numbers of CPU cores, or different interconnect bandwidths.

[0135] The first device can send its own computing power information and target optimization instructions to the second device, for example, through API calls, message queues, configuration synchronization services, or model repository metadata fields.

[0136] After receiving the above information, the second device first parses its own computing power information and compares it with the computing power information of the first device to calculate the computing power difference between the two. This computing power difference can be reflected in performance scaling factors (e.g., the speedup of general matrix multiplication by the A-type GPU of the first device is about 1.8 times compared to the B-type GPU of the second device), resource constraint differences (e.g., the second device has less video memory and needs to reduce the batch size), or functional support differences (e.g., the first device uses specific Tensor Core instructions that are only supported by A100), etc.

[0137] The second device can adapt the target optimization instructions based on the difference in computing power. If the second device has stronger computing power, it can further aggressively optimize the target optimization instructions (e.g., increase the parallel granularity); if the second device has weaker computing power or lacks certain hardware features, it can downgrade or replace the target optimization instructions (e.g., fall back to a general kernel implementation). The upper-layer training framework of the second device responds to the adapted target optimization instructions by adjusting the training configuration parameters associated with the target optimization instructions to improve the training performance of the target model deployed on the second device.

[0138] In other embodiments, a performance normalization model, a hardware abstraction layer mapping table, or a multi-device collaborative optimization protocol may be introduced to further enhance the robustness and compatibility of instruction migration. The specific implementation method is not limited here.

[0139] Understandably, by transferring optimization experience from model training across devices and making adaptive adjustments based on local computing power differences, the migration and generalization of training optimization strategies are achieved. This avoids repeating the entire performance analysis and tuning process on each new device, significantly improving model deployment efficiency and resource utilization.

[0140] Furthermore, in some embodiments, the above method further includes: after updating the pre-built or trained operator performance model according to the execution information, obtaining the updated operator performance model, and sending the updated operator performance model to the second device; wherein the updated operator performance model is used to iteratively train the target model during the training process of the target model by the second device.

[0141] The method for updating the pre-built or trained operator performance model based on the execution information is the same as described above and will not be repeated here.

[0142] After completing the model update, the first device exports the updated operator performance model in a portable format (such as PMML (Predictive Model Markup Language), a custom binary file, or a serialized object) and sends it to the second device via a network interface, shared storage, model repository, or message queue. This transmission can be synchronized with the target optimization instructions and computing power information, or it can be triggered independently, for example, pushed as metadata at the end of the training task.

[0143] After receiving the updated operator performance model, the second device loads it into the optimization controller of the local training framework, replacing or fusing the original model. During subsequent iterative training of the target model, the second device directly calls the updated operator performance model to predict the execution time of each computational operator, identify global critical paths, or assist in generating latency optimization instructions suitable for the local environment.

[0144] For example, when the first device trains the target model Transformer on a GPU of type A, actual testing reveals that a certain general matrix multiplication operator experiences higher-than-expected latency due to memory bandwidth contention. Therefore, the corresponding entry in the operator performance model is updated, resulting in an updated operator performance model. This updated model is then sent to a second device using a GPU of type B. After loading the updated model, the second device can more accurately predict operator latency when training the same target model Transformer, thereby identifying potential bottlenecks in advance and applying adaptive optimizations.

[0145] In other embodiments, the second device may also perform secondary fine-tuning on the received model in conjunction with local execution feedback to achieve collaborative evolution; the specific implementation method is not limited here.

[0146] Understandably, by sharing the operator performance model calibrated through actual operation, performance knowledge can be transferred across devices, enabling the second device to obtain high-precision performance prediction capabilities without going through a complete observation-analysis-tuning cycle, thus significantly improving training efficiency and resource utilization.

[0147] In an example embodiment of this disclosure, identifying the target operator constituting the training iteration delay among the computational operators included in the global critical path includes: obtaining the total computation time of the global critical path during the current batch training process, and obtaining the historical computation time of the same global critical path of the target model during the historical batch training process; if the total computation time is greater than the historical computation time, then identifying a new target operator constituting the training iteration delay among the computational operators included in the global critical path; the target operator during the current batch training process is different from the target operator during the historical batch training process.

[0148] The historical batches can include at least one batch preceding the current batch. For example, a historical batch can be the previous batch or any batch preceding the current batch, or multiple batches preceding the current batch. If the historical batches are multiple batches preceding the current batch, then these multiple batches can be consecutive batches or non-consecutive discrete batches.

[0149] Historical computation time refers to the total computation time recorded for the same global critical path during historical batch training. This historical computation time can be stored in the training controller's time-series database, sliding window cache, or model metadata, and is used to characterize the typical latency level of the critical path in a steady state.

[0150] For example, historical computation time can be the total computation time calculated on the same global critical path during the previous batch of training, or it can be the average, minimum, or weighted exponential moving average of the total computation time calculated on the same global critical path during the most recent N batches of training, in order to balance stability and sensitivity to changes.

[0151] After determining the global critical path and its total computation time in the current batch training process, the total computation time in the current batch training process can be compared with the historical computation time of the same global critical path in previous batches. If the current total computation time is greater than the historical computation time in previous batch training processes, it indicates that the training performance has abnormally degraded or the bottleneck has shifted, and the target operator in the historical batch training process of this global critical path may no longer be the main source of latency.

[0152] Therefore, during the current batch training process, all computation operators on the same global critical path can be retraced. Based on the updated computation time, latency contribution (i.e., the proportion of a single operator's time to the total path time), and the stability of their appearance in consecutive batches, a new target operator that has a dominant influence on the latency of the current iteration can be re-identified. This new target operator is different from the target operators in the historical batch training process.

[0153] For example, the target operator in the historical batch training process was the gradient synchronization communication operation, and its latency had been reduced to 1.2 milliseconds after pre-order optimization. However, in the current batch training process, due to the change in the distribution of input data, the computation time of the general matrix multiplication in a certain feedforward network layer in the same global critical path suddenly increased to 5.8 milliseconds, making the general matrix multiplication operator a new performance bottleneck in the current batch training process of the global critical path. Therefore, it was re-identified as the target operator.

[0154] In other embodiments, a performance mutation detection algorithm (such as a sliding t-test algorithm) or a confidence interval may be introduced to determine whether re-identification is triggered. The specific implementation method is not limited here.

[0155] Understandably, by obtaining the total computation time of the global critical path during the current batch training process and comparing it with the historical computation time of the same global critical path in previous batches, it is possible to promptly detect performance degradation or bottleneck drift in scenarios where training is dynamically changing. When it is detected that the current total computation time of the same global critical path exceeds the historical computation time of previous batches, it indicates that the original optimization strategy for that global critical path may have failed, or that new runtime disturbances have occurred (such as sudden changes in input data distribution, increased competition for hardware resources, etc.). Therefore, the latency contribution of each computation operator in that global critical path can be re-analyzed, the new target operator that is currently dominating the training iteration latency can be identified, and targeted latency optimization instructions can be generated. This achieves closed-loop perception and adaptive response to dynamic changes in training performance, avoiding the failure of optimization strategies due to environmental disturbances, data drift, or hardware fluctuations.

[0156] In some embodiments, the cross-device platform migration process includes: after the first device completes training of the target model on a first graphics processor (e.g., GPU A), it exports an operator performance model updated with underlying kernel execution information; the second device loads the target model and the updated operator performance model on a second graphics processor cluster (e.g., GPU B cluster); the computing power difference between the first and second devices can be determined based on their computing power information, and the startup configuration parameters of the underlying kernel (including grid dimension, block dimension, shared memory allocation, etc.) can be automatically adjusted according to the computing power difference; thus, the second device can obtain near-optimal computing performance during training without manual readjustment.

[0157] Figure 4 The schematic diagram illustrates a flow chart of a model training method according to some other embodiments of the present disclosure. Reference Figure 4 As shown, the model training method is performed by a second device and may include the following steps: Step S410: Receive the computing power information and target optimization instructions sent by the first device after the target model training is completed; the target optimization instructions are generated based on the delay optimization instructions of the target model during the iterative training process, and the delay optimization instructions are obtained according to the above model training method. Step S420: Obtain the computing power information of the second device, and determine the computing power difference between the computing power information of the second device and the computing power information of the first device during the current batch training process of the target model deployed in the second device; Step S430: Based on the difference in computing power and the target optimization instructions, adjust the training configuration parameters of the upper-layer training framework of the processor of the second device so as to apply the adjusted training configuration parameters in the subsequent batch training process of the target model.

[0158] According to the model training method in this example embodiment, when deploying the target model, the second device receives computing power information and target optimization instructions sent by the first device after training is completed. The target optimization instructions are based on latency optimization experience generated and aggregated by the first device through real underlying kernel execution information during iterative training. The second device, combining its own computing power information, determines the computing power difference between itself and the first device, and adjusts the target optimization instructions accordingly. This modifies the training configuration parameters corresponding to the target operator in its upper-layer processor training framework, applying the adjusted training configuration parameters in subsequent batches of target model training. In other words, the second device can reuse optimization strategies validated on the first device without repeating the complete performance observation, critical path analysis, and optimization decision-making process. It can also dynamically adapt to its own hardware capabilities, enabling the target model training process on the second device to run in near-optimal configuration from the initial batch. This effectively avoids performance degradation or configuration failure caused by hardware heterogeneity, thereby significantly improving the training performance and deployment efficiency of the target model on the second device.

[0159] The model training method in this example embodiment will be further explained below.

[0160] In step S410, the computing power information and target optimization instructions of the first device sent by the first device after the target model training is completed are received; the target optimization instructions are generated based on the delay optimization instructions of the target model during the iterative training process, and the delay optimization instructions are obtained according to the above model training method.

[0161] In step S420, the computing power information of the second device is obtained, and during the current batch training process of the target model deployed in the second device, the computing power difference between the computing power information of the second device and the computing power information of the first device is determined.

[0162] In some embodiments, the computing power information of the second device and the computing power information of the first device can be differentially processed to obtain the computing power difference between the two.

[0163] In other embodiments, the ratio between the computing power information of the second device and the computing power information of the first device can also be calculated to obtain the computing power difference between the two.

[0164] In other embodiments, other methods may be used to determine the difference in computing power between the two, which are not limited here.

[0165] In step S430, based on the difference in computing power and the target optimization instructions, the training configuration parameters of the upper-layer training framework of the processor of the second device are adjusted so that the adjusted training configuration parameters can be applied in subsequent batch training of the target model.

[0166] Specifically, the target optimization instructions are updated based on the difference in computing power to obtain updated target optimization instructions; the updated target optimization instructions are matched with the computing power information of the second device; in response to the updated target optimization instructions, the training configuration parameters of the upper-layer training framework of the processor of the second device are adjusted.

[0167] If the difference in computing power indicates that the second device has a stronger computing power than the first device, then the optimization intensity can be increased based on the optimization instructions for that objective. For example, the bucket size for gradient synchronization communication can be increased from 256 megabytes to 384 megabytes, or the version of the high-performance attention computing kernel can be upgraded from the second generation to the third generation.

[0168] If the difference in computing power indicates that the second device's computing power is weaker than the first device's, or that the second device lacks specific hardware features (such as V100 not supporting FP8 or Tensor Core sparse computing), then the target optimization instructions will be downgraded or replaced. This could involve reverting to a generic cuBLAS implementation, disabling high-order fusion strategies, or reducing batch size. This update process can be completed automatically based on a pre-built hardware capability mapping table, kernel compatibility rule base, or runtime detection results, ensuring that the updated target optimization instructions strictly match the actual capabilities of the second device.

[0169] The second device can parse the updated target optimization instructions into specific training configuration parameters and inject them into the scheduler, compiler, or communication backend of the upper-layer training framework (such as PyTorch, DeepSpeed, or Megatron) of the second device processor.

[0170] For example, setting environment variables, modifying operator registration priorities, reloading kernel selection strategies, or adjusting the bucket configuration for distributed training. These adjustments enable the target model to fully utilize local hardware resources in subsequent training iterations, avoiding invalid or conflicting configurations, thereby significantly improving training throughput, reducing iteration latency, and enhancing system stability.

[0171] In other embodiments, the updated instructions can be further verified by combining local micro-benchmark tests, or a feedback mechanism can be introduced to continuously optimize the adaptation strategy. The specific implementation method is not limited here.

[0172] Furthermore, in some embodiments, the above method further includes: receiving an updated operator performance model sent by a first device; the updated operator performance model is obtained by the first device updating a pre-built or trained operator performance model based on the execution information of the processor's underlying kernel during the training process of the target model; loading the updated operator performance model, and iteratively training the target model deployed in the second device based on the updated operator performance model.

[0173] During subsequent iterative training of the target model, the second device can directly call the updated operator performance model to predict the execution time of each computational operator, identify the global critical path of the current batch, evaluate potential bottleneck operators, or assist in generating latency optimization instructions suitable for the local environment.

[0174] For example, without performing a complete performance analysis, the second device can predict whether the latency of a certain attention operator is abnormal under the current batch size based on the updated operator performance model, thereby triggering the optimization strategy in advance.

[0175] Understandably, by reusing operator performance models calibrated with real-world data, the second device can achieve high-fidelity performance prediction capabilities without going through a complete data acquisition, modeling, and validation cycle, significantly improving the efficiency of adaptive optimization during the training process.

[0176] Understandably, after the second device loads the updated operator performance model, it uses this updated operator performance model as the pre-trained operator performance model. During subsequent batch training, it can obtain the execution information of the underlying kernel corresponding to the multiple computational operators of the target model executed by the second device's processor, and update the pre-trained operator performance model based on this execution information to obtain the updated operator performance model. Using the updated operator performance model, it determines the global critical path with the longest total computation time in the current batch training process of the target model, and identifies the target operators that constitute the training iteration delay within the global critical path. It generates delay optimization instructions for the target operators and adjusts the training configuration parameters corresponding to the target operators in the upper-layer training framework of the processor in response to these delay optimization instructions, thereby improving the training performance of the target model in subsequent batch training processes.

[0177] It should be noted that the second device loads the updated operator performance model and uses it as the pre-trained operator performance model. The model training process, including updating the pre-trained operator performance model based on the execution information, determining the global critical path, identifying the target operator, and adjusting the training configuration parameters, is the same as or similar to the model training process of the first device. Therefore, it will not be elaborated here.

[0178] It should be noted that although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0179] Furthermore, in this example embodiment, a model training apparatus is also provided. (Refer to...) Figure 5As shown, the model training device 500 includes: an execution information acquisition module 510, a training iteration delay analysis module 520, and a first training configuration adjustment module 530. Wherein: The execution information acquisition module 510 is used to acquire the execution information of the underlying kernel when the processor executes multiple computation operators of the target model during the current batch training process of the target model. The training iteration delay analysis module 520 is used to determine the global critical path of the target model related to the total computation time in the current batch training process based on the execution information, and to identify the target operators that constitute the training iteration delay among the computation operators contained in the global critical path. The first training configuration adjustment module 530 is used to generate latency optimization instructions for the target operator and adjust the training configuration parameters corresponding to the target operator in the upper-layer training framework of the processor in response to the latency optimization instructions, so as to apply the adjusted training configuration parameters in subsequent batch training of the target model.

[0180] In some example embodiments of this disclosure, based on the foregoing scheme, the training iteration delay analysis module 520 is configured as follows: The pre-built or trained operator performance model is updated based on the execution information to obtain the updated operator performance model; By using the updated operator performance model, the global critical path with the longest total computation time for the target model during the current batch training process is determined.

[0181] In some example embodiments of this disclosure, based on the foregoing scheme, the training iteration delay analysis module 520 is configured as follows: The execution information is mapped to obtain the operator performance, and the operator-related features in the current batch training process are obtained; the operator-related features include at least one of the operator input shape, scheduling parameters and hardware features; Construct a mapping relationship between operator performance and operator-related features, and update the pre-constructed or trained operator performance model based on the mapping relationship to obtain the updated operator performance model.

[0182] In some example embodiments of this disclosure, based on the foregoing scheme, the training iteration delay analysis module 520 is configured as follows: By using the updated operator performance model, the computation time of each computation operator in the current batch training process of the target model is determined; Based on the computation time of each computation operator, the global critical path with the longest total computation time is determined.

[0183] In some example embodiments of this disclosure, based on the foregoing scheme, the training iteration delay analysis module 520 is configured as follows: Obtain the training computation graph of the target model, which includes the sequential dependencies between various computation operators; Based on the sequential dependencies between computation operators and the computation time of each computation operator, the global critical path with the longest total computation time is determined.

[0184] In some example embodiments of this disclosure, based on the foregoing scheme, the training iteration delay analysis module 520 is configured as follows: Based on the sequential dependencies between the computation operators, determine the global candidate path for the target model in the current batch training process; Based on the computation time of each computation operator on the global candidate path, the total computation time of the global candidate path is determined, and the global candidate path with the longest total computation time is taken as the global critical path.

[0185] In some example embodiments of this disclosure, the first training configuration adjustment module 530 is configured, based on the foregoing scheme, to generate target optimization instructions based on the delay optimization instructions in the iterative training process after the target model training is completed; The device also includes a sending module configured to send the computing power information of the first device and the target optimization instructions to the second device; wherein, [dlr11.1] the computing power information of the first device is used to determine the computing power difference between the computing power information of the first device and the computing power information of the second device during the training of the target model on the second device, and the computing power difference is used to adjust the training configuration parameters of the upper-layer training framework of the processor of the second device in combination with the target optimization instructions.

[0186] In some example embodiments of this disclosure, based on the foregoing scheme, the sending module is configured as follows: After updating the pre-built or trained operator performance model based on the execution information, the updated operator performance model is obtained and sent to the second device; The updated operator performance model is used to iteratively train the target model during the training process of the target model on the second device.

[0187] In some example embodiments of this disclosure, based on the foregoing scheme, the training iteration delay analysis module 520 is configured as follows: Get the total computation time of the global critical path during the current batch training process, and get the historical computation time of the target model on the same global critical path during the historical batch training process; If the total computation time is greater than the historical computation time, then identify the new target operator that constitutes the training iteration delay among the computation operators contained in the global critical path; the target operator in the current batch training process is different from the target operator in the historical batch training process.

[0188] In some example embodiments of this disclosure, based on the foregoing scheme, the training iteration delay analysis module 520 is configured as follows: For each computation operator contained in the global critical path, obtain the computation time of the computation operator and determine the dependency path to which the computation operator belongs; Based on the computation time of each computation operator and its dependent path, the target operators that constitute the training iteration delay in the global critical path are determined.

[0189] Furthermore, in this example embodiment, another model training apparatus is provided, which is performed by a second device. (See also...) Figure 6 As shown, the model training device 500 includes: a receiving module 610, a computing power difference calculation module 620, and a second training configuration adjustment module 630. Wherein: The receiving module 610 is used to receive the computing power information and target optimization instructions of the first device after the target model training is completed; the target optimization instructions are generated based on the delay optimization instructions of the target model during the iterative training process, and the delay optimization instructions are obtained according to the above-mentioned model training method. The computing power difference calculation module 620 is used to obtain the computing power information of the second device and determine the computing power difference between the computing power information of the second device and the computing power information of the first device during the current batch training process of the target model deployed in the second device. The second training configuration adjustment module 630 is used to adjust the training configuration parameters of the upper-layer training framework of the processor of the second device based on the difference in computing power and the target optimization instructions, so as to apply the adjusted training configuration parameters in the subsequent batch training process of the target model.

[0190] In some example embodiments of this disclosure, based on the foregoing scheme, the second training configuration adjustment module 630 is configured as follows: The target optimization instruction is updated based on the difference in computing power to obtain the updated target optimization instruction; the updated target optimization instruction is matched with the computing power information of the second device. In response to the updated target optimization instructions, the training configuration parameters of the upper-layer training framework of the second device's processor are adjusted.

[0191] In some example embodiments of this disclosure, based on the foregoing scheme, the receiving module 610 is configured to: receive the updated operator performance model sent by the first device; the updated operator performance model is obtained by the first device updating the pre-built or trained operator performance model according to the execution information of the processor's underlying kernel during the training process of the target model; The second training configuration adjustment module 630 is configured to: load the updated operator performance model and iteratively train the target model deployed in the second device based on the updated operator performance model.

[0192] The specific details of each module of the above model training device have been described in detail in the corresponding model training methods, so they will not be repeated here.

[0193] It should be noted that although several modules or units of the model training apparatus have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0194] Furthermore, in an exemplary embodiment of this disclosure, an electronic device capable of implementing the above-described model training method is also provided.

[0195] Those skilled in the art will understand that various aspects of this disclosure can be implemented as a system, method, or program product. Therefore, various aspects of this disclosure can be embodied in the following forms: a completely hardware embodiment, a completely software embodiment (including firmware, microcode, etc.), or an embodiment combining hardware and software aspects, collectively referred to herein as a "circuit," "module," or "system."

[0196] The following reference Figure 7 To describe an electronic device 700 according to such an embodiment of the present disclosure. Figure 7 The electronic device 700 shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein. It is understood that the electronic device may be either the first device or the second device in the aforementioned model training method.

[0197] like Figure 7 As shown, the electronic device 700 is manifested in the form of a general-purpose computing device. The components of the electronic device 700 may include, but are not limited to: at least one processing unit 710, at least one storage unit 720, a bus 730 connecting different system components (including storage unit 720 and processing unit 710), and a display unit 740.

[0198] The storage unit stores program code that can be executed by the processing unit 710, causing the processing unit 710 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure. For example, the processing unit 710 can perform actions such as... Figure 1In step S110, during the current batch training of the target model, the execution information of the underlying kernel corresponding to the processor executing multiple computation operators of the target model is obtained; in step S120, based on the execution information, the global critical path related to the total computation time of the target model in the current batch training is determined, and the target operators constituting the training iteration delay among the computation operators contained in the global critical path are identified; in step S130, a delay optimization instruction for the target operator is generated, and the training configuration parameters corresponding to the target operator in the upper-level training framework of the processor are adjusted in response to the delay optimization instruction, so as to apply the adjusted training configuration parameters in the subsequent batch training of the target model.

[0199] Storage unit 720 may include readable media in the form of volatile storage units, such as random access memory (RAM) 721 and / or cache memory 722, and may further include read-only memory (ROM) 723.

[0200] The storage unit 720 may also include a program / utility 724 having a set (at least one) of program modules 725, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0201] Bus 730 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0202] Electronic device 700 can also communicate with one or more external devices 770 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 700, and / or with any device that enables electronic device 700 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 750. Furthermore, electronic device 700 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 760. As shown, network adapter 760 communicates with other modules of electronic device 700 via bus 730. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 700, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0203] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0204] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the methods described above is stored. In some possible embodiments, various aspects of this disclosure may also be implemented as a program product including program code that, when the program product is run on a terminal device, causes the terminal device to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure.

[0205] refer to Figure 8 As shown, a program product 800 for implementing the above-described model training method according to an embodiment of the present disclosure is described. This product may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0206] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0207] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0208] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0209] Program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0210] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of this disclosure and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0211] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, touch terminal, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0212] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0213] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A model training method, characterized in that, include: During the current batch training of the target model, the execution information of the underlying kernel corresponding to the processor executing multiple computational operators of the target model is obtained; Based on the execution information, determine the global critical path of the target model related to the total computation time during the current batch training process, and identify the target operator that constitutes the training iteration delay among the computation operators contained in the global critical path; Generate a latency optimization instruction for the target operator, and adjust the training configuration parameters corresponding to the target operator in the upper-layer training framework of the processor in response to the latency optimization instruction, so as to apply the adjusted training configuration parameters in subsequent batch training of the target model.

2. The model training method according to claim 1, characterized in that, The step of determining the global critical path of the target model related to the total computation time during the current batch training process based on the execution information includes: The pre-built or trained operator performance model is updated based on the execution information to obtain the updated operator performance model; The updated operator performance model is used to determine the global critical path with the longest total computation time for the target model during the current batch training process.

3. The model training method according to claim 2, characterized in that, The step of updating the pre-built or trained operator performance model based on the execution information to obtain the updated operator performance model includes: The execution information is mapped to obtain operator performance, and operator-related features in the current batch training process are obtained; the operator-related features include at least one of operator input shape, scheduling parameters, and hardware features; A mapping relationship is constructed between the operator performance and the operator-related features, and the pre-constructed or trained operator performance model is updated based on the mapping relationship to obtain the updated operator performance model.

4. The model training method according to claim 2, characterized in that, The step of determining the global critical path with the longest total computation time for the target model during the current batch training process using the updated operator performance model includes: The updated operator performance model is used to determine the computation time of each computation operator in the current batch training process of the target model. Based on the computation time of each of the aforementioned computation operators, the global critical path with the longest total computation time is determined.

5. The model training method according to claim 4, characterized in that, The process of determining the global critical path with the longest total computation time based on the computation time of each of the aforementioned computation operators includes: Obtain the training computation graph of the target model, wherein the training computation graph includes the sequential dependencies between the computation operators; Based on the sequential dependencies between the computation operators and the computation time of each computation operator, the global critical path with the longest total computation time is determined.

6. The model training method according to claim 5, characterized in that, The process of determining the global critical path with the longest total computation time based on the sequential dependencies between the computation operators and the computation time of each computation operator includes: Based on the sequential dependencies between the computational operators, the global candidate path of the target model in the current batch training process is determined; Based on the computation time of each computation operator on the global candidate path, the total computation time of the global candidate path is determined, and the global candidate path with the longest total computation time is taken as the global critical path.

7. The model training method according to claim 1, characterized in that, The method is performed by a first device, and the method further includes: After the target model is trained, a target optimization instruction is generated based on each of the delay optimization instructions in the iterative training process; The computing power information of the first device and the target optimization instructions are sent to the second device; The computing power information of the first device is used to determine the computing power difference between the computing power information of the first device and the computing power information of the second device during the training of the target model by the second device. The computing power difference is used to adjust the training configuration parameters of the upper-layer training framework of the processor of the second device in combination with the target optimization instructions.

8. The model training method according to claim 1, characterized in that, The method further includes: After updating the pre-built or trained operator performance model according to the execution information, the updated operator performance model is obtained and sent to the second device; The updated operator performance model is used to iteratively train the target model during the training process of the target model by the second device.

9. The model training method according to any one of claims 1 to 8, characterized in that, The identification of the target operators constituting the training iteration delay among the computational operators included in the global critical path includes: Obtain the total computation time of the global critical path during the current batch training process, and obtain the historical computation time of the target model on the same global critical path during historical batch training processes; If the total computation time is greater than the historical computation time, then the target operator constituting the training iteration delay is identified among the computation operators included in the global critical path; the target operator in the current batch training process is different from the target operator in the historical batch training process.

10. The model training method according to any one of claims 1 to 8, characterized in that, The identification of the target operators constituting the training iteration delay among the computational operators included in the global critical path includes: For each computation operator contained in the global critical path, obtain the computation time of the computation operator and determine the dependency path to which the computation operator belongs; Based on the computation time of each computation operator and the dependency path to which it belongs, the target operator constituting the training iteration delay is determined among the computation operators included in the global critical path.

11. A model training method, characterized in that, The method is performed by a second device, and the method includes: The system receives computing power information and target optimization instructions sent by the first device after the target model training is completed; the target optimization instructions are generated based on the delay optimization instructions of the target model during iterative training, and the delay optimization instructions are obtained by the model training method according to any one of claims 1 to 10. Obtain the computing power information of the second device, and during the current batch training process of the target model deployed in the second device, determine the computing power difference between the computing power information of the second device and the computing power information of the first device; Based on the computing power difference and the target optimization instructions, the training configuration parameters of the upper-layer training framework of the processor of the second device are adjusted so that the adjusted training configuration parameters can be applied in subsequent batch training of the target model.

12. The model training method according to claim 11, characterized in that, The step of adjusting the training configuration parameters of the upper-layer training framework of the processor of the second device based on the computing power difference and the target optimization instructions includes: The target optimization instruction is updated based on the computing power difference to obtain an updated target optimization instruction; the updated target optimization instruction is matched with the computing power information of the second device. In response to the updated target optimization instructions, the training configuration parameters of the processor upper-layer training framework of the second device are adjusted.

13. The model training method according to claim 11, characterized in that, The method further includes: The system receives an updated operator performance model sent by the first device. The updated operator performance model is obtained by the first device updating a pre-built or trained operator performance model based on the execution information of the processor's underlying kernel during the training of the target model. The updated operator performance model is loaded, and the target model deployed in the second device is iteratively trained based on the updated operator performance model.

14. A model training device, characterized in that, include: The execution information acquisition module is used to acquire the execution information of the underlying kernel corresponding to the processor executing multiple computational operators of the target model during the current batch training process of the target model. The training iteration delay analysis module is used to determine the global critical path of the target model related to the total computation time during the current batch training process based on the execution information, and to identify the target operators that constitute the training iteration delay among the computation operators contained in the global critical path. The first training configuration adjustment module is used to generate a latency optimization instruction for the target operator, and adjust the training configuration parameters corresponding to the target operator in the upper-layer training framework of the processor in response to the latency optimization instruction, so as to apply the adjusted training configuration parameters in subsequent batch training of the target model.

15. A model training device, characterized in that, Performed by a second device, the apparatus comprising: A receiving module is configured to receive computing power information and target optimization instructions sent by the first device after the target model training is completed; the target optimization instructions are generated based on the delay optimization instructions of the target model during the iterative training process, and the delay optimization instructions are obtained by the model training method according to any one of claims 1 to 10. The computing power difference calculation module is used to obtain the computing power information of the second device and determine the computing power difference between the computing power information of the second device and the computing power information of the first device during the current batch training process of the target model deployed in the second device. The second training configuration adjustment module is used to adjust the training configuration parameters of the upper-layer training framework of the processor of the second device based on the computing power difference and the target optimization instructions, so as to apply the adjusted training configuration parameters in the subsequent batch training process of the target model.

16. An electronic device, characterized in that, include: processor; as well as A memory storing computer-readable instructions that, when executed by the processor, implement the model training method as described in any one of claims 1 to 13.

17. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the model training method as described in any one of claims 1 to 13.

Images