Model running performance evaluation method, performance evaluation model training method and device

Abstract

The present disclosure provides a model running performance evaluation method, a performance evaluation model training method and device. The method comprises: obtaining a target model and configuration parameters running in a to-be-evaluated running environment; inputting the configuration parameters into a performance evaluation model, wherein the performance evaluation model comprises a plurality of performance modeling layers and a compensation layer connected in sequence, each performance modeling layer corresponds to a plurality of logical processing modules executed in sequence in the running process of the target model, and the plurality of performance modeling layers are connected in sequence according to the execution order of the corresponding logical processing modules; inputting the configuration parameters into the plurality of performance modeling layers for step-by-step feature processing; inputting the intermediate feature vector output by the last performance modeling layer into the compensation layer to compensate for the cumulative error generated by the step-by-step feature processing of the plurality of performance modeling layers, and obtain the performance evaluation result of the target model in the to-be-evaluated running environment. The present disclosure can accurately evaluate the performance of the model deployed in the to-be-evaluated running environment in the case of missing details of the running environment.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence, and in particular to a method for evaluating the performance of a model, a method for training the performance evaluation model, and an apparatus. Background Technology

[0002] To evaluate the performance of AI (Artificial Intelligence) models in different operating environments, it is usually necessary to predict the performance of the AI ​​model in the operating environment to be evaluated, so as to guide the hardware selection or deployment planning for the corresponding operating environment.

[0003] The performance evaluation relies on two methods: one is to estimate the theoretical peak performance based on the parameters of the operating environment to be evaluated, and the other is to obtain performance data by performing instruction-level or microarchitecture-level simulation based on the details of the underlying hardware architecture.

[0004] However, both of the above methods rely on the hardware architecture details of the operating environment to be evaluated. In real-world applications, these hardware architecture details may be in an undetermined or confidential state, making performance evaluation difficult or resulting in low reliability of the evaluation results. Summary of the Invention

[0005] This disclosure provides a method for evaluating the performance of a model, a method for training a performance evaluation model, and an apparatus that can accurately evaluate the performance of an artificial intelligence model deployed in the environment to be evaluated, even when details of the operating environment (such as hardware architecture details) are missing.

[0006] According to one aspect of this disclosure, a method for evaluating the performance of a model is provided, comprising: acquiring configuration parameters of a target model and a runtime environment to be evaluated; inputting the configuration parameters into a performance evaluation model, wherein the performance evaluation model includes multiple sequentially connected performance modeling layers and a compensation layer, each performance modeling layer corresponding to multiple logical processing modules executed sequentially during the operation of the target model, the multiple performance modeling layers being connected sequentially according to the execution order of the corresponding logical processing modules; inputting the configuration parameters into the multiple performance modeling layers for step-by-step feature processing, wherein intermediate feature vectors output by the preceding performance modeling layer are input into the following performance modeling layer, until an intermediate feature vector is obtained from the final performance modeling layer; inputting the intermediate feature vectors output by the final performance modeling layer into the compensation layer to compensate for the cumulative error generated by the step-by-step feature processing of the multiple performance modeling layers, thereby obtaining the performance evaluation result of the target model in the runtime environment to be evaluated.

[0007] Optionally, the configuration parameters include common configuration parameters and hierarchical configuration parameters corresponding to each of the performance modeling layers; inputting the configuration parameters into the multiple performance modeling layers for step-by-step feature processing includes: performing feature processing on the common configuration parameters to obtain a common feature vector; performing feature processing on the hierarchical configuration parameters corresponding to the starting performance modeling layer to obtain a starting layer configuration feature vector; determining the input feature vector of the starting performance modeling layer based on the common feature vector and the starting layer configuration feature vector; and inputting the input feature vector of the starting performance modeling layer into the starting performance modeling layer for feature transformation to obtain the intermediate feature vector output by the starting performance modeling layer.

[0008] Optionally, the step of inputting the configuration parameters into the plurality of performance modeling layers for step-by-step feature processing further includes: for the current performance modeling layer after the initial performance modeling layer, determining the hierarchical configuration parameters corresponding to the current performance modeling layer from the configuration parameters; performing feature processing on the hierarchical configuration parameters corresponding to the current performance modeling layer to obtain the current layer configuration feature vector; determining the input feature vector of the current performance modeling layer based on the intermediate feature vector output by the previous performance modeling layer and the current layer configuration feature vector; inputting the input feature vector of the current performance modeling layer into the current performance modeling layer for feature transformation to obtain the intermediate feature vector output by the current performance modeling layer; and sequentially performing the above processing on each performance modeling layer after the initial performance modeling layer until the intermediate feature vector output by the final performance modeling layer is obtained.

[0009] Optionally, the step of inputting the configuration parameters into the plurality of performance modeling layers for step-by-step feature processing further includes: performing performance mapping on the intermediate feature vectors output by each performance modeling layer to obtain the local performance evaluation results corresponding to each performance modeling layer; wherein, each of the local performance evaluation results represents the operating performance of the corresponding logic processing module in the operating environment to be evaluated.

[0010] Optionally, the step of performing feature processing on the common configuration parameters to obtain a common feature vector includes: determining numerical common configuration parameters and categorical common configuration parameters from the common configuration parameters; normalizing the numerical common configuration parameters and performing feature mapping on the normalized numerical common configuration parameters to obtain a common numerical feature vector; performing discrete value encoding on the categorical common configuration parameters and embedding the discrete value encoded categorical common configuration parameters to obtain a common category feature vector; and concatenating the common numerical feature vector and the common category feature vector to obtain the common feature vector.

[0011] Optionally, the feature processing of the hierarchical configuration parameters corresponding to the current performance modeling layer includes: determining numerical hierarchical configuration parameters and categorical hierarchical configuration parameters from the hierarchical configuration parameters; normalizing the numerical hierarchical configuration parameters and performing feature mapping on the normalized numerical hierarchical configuration parameters to obtain a hierarchical numerical feature vector; performing discrete value encoding on the categorical hierarchical configuration parameters and embedding the discrete value encoded categorical hierarchical configuration parameters to obtain a hierarchical category feature vector; concatenating the hierarchical numerical feature vector and the hierarchical category feature vector to obtain the current layer configuration feature vector; and concatenating the intermediate feature vector output by the previous performance modeling layer and the current layer configuration feature vector to obtain the input feature vector of the current performance modeling layer.

[0012] According to one aspect of this disclosure, a method for training a performance evaluation model is provided. The initial performance evaluation model to be trained includes multiple sequentially connected performance modeling layers and compensation layers. Each performance modeling layer corresponds to multiple logical processing modules executed sequentially during the execution of the target model. The multiple performance modeling layers are connected sequentially according to the execution order of the corresponding logical processing modules. The method includes: obtaining sample configuration parameters of the target model running in a sample runtime environment, the end-to-end performance label corresponding to the target model, and the local performance labels corresponding to each logical processing module; inputting the sample configuration parameters into the multiple performance modeling layers for step-by-step feature processing to obtain the predicted local performance evaluation results corresponding to each performance modeling layer, and obtaining the intermediate feature vector output by the final-level performance modeling layer; and processing the final-level... The intermediate feature vector output by the performance modeling layer is input into the compensation layer to compensate for the cumulative error generated by the step-by-step feature processing of the multiple performance modeling layers, thereby obtaining the prediction end-to-end performance evaluation result. For each performance modeling layer, the corresponding local prediction loss is determined based on the prediction local performance evaluation result and the local performance label of the logic processing module corresponding to the performance modeling layer. The end-to-end prediction loss is determined based on the prediction end-to-end performance evaluation result and the end-to-end performance label. The joint prediction loss is determined based on each local prediction loss and the end-to-end prediction loss, and the model parameters of the initial performance evaluation model are updated based on the joint prediction loss until the training stopping condition is met, thereby obtaining the performance evaluation model.

[0013] Optionally, obtaining the end-to-end performance label corresponding to the target model and the local performance labels corresponding to each of the logical processing modules includes: deploying the target model in the sample running environment, measuring the end-to-end performance of the target model when it completes a preset running task, and obtaining measured end-to-end performance values; determining the end-to-end performance label based on the measured end-to-end performance values; deploying the logical processing modules corresponding to each performance modeling layer in the sample running environment, measuring the local performance of each logical processing module when it completes its corresponding processing task, and obtaining measured local performance values ​​corresponding to each logical processing module; and determining the local performance label corresponding to each logical processing module based on the measured local performance values.

[0014] Optionally, the step of inputting the sample configuration parameters into the plurality of performance modeling layers for step-by-step feature processing includes: determining sample common configuration parameters and sample layer configuration parameters corresponding to each of the performance modeling layers from the sample configuration parameters; performing feature processing on the sample common configuration parameters to obtain a sample common feature vector; performing feature processing on the sample layer configuration parameters corresponding to the starting performance modeling layer to obtain a sample starting layer configuration feature vector; determining the sample input feature vector of the starting performance modeling layer based on the sample common feature vector and the sample starting layer configuration feature vector; and inputting the sample input feature vector into the starting performance modeling layer for feature transformation to obtain... The intermediate feature vector output by the initial performance modeling layer is obtained; for the current performance modeling layer after the initial performance modeling layer, feature processing is performed on the sample level configuration parameters corresponding to the current performance modeling layer to obtain the current layer sample configuration feature vector; based on the intermediate feature vector output by the previous performance modeling layer and the current layer sample configuration feature vector, the sample input feature vector of the current performance modeling layer is determined; the sample input feature vector of the current performance modeling layer is input into the current performance modeling layer for feature transformation to obtain the intermediate feature vector output by the current performance modeling layer, and the above feature processing is performed sequentially until the intermediate feature vector output by the final performance modeling layer is obtained.

[0015] Optionally, determining the joint prediction loss based on each of the local prediction losses and the end-to-end prediction losses includes: calculating the log-mean square error between each predicted local performance evaluation result and the corresponding local performance label to obtain each of the local prediction losses; calculating the log-mean square error between the predicted end-to-end performance evaluation result and the end-to-end performance label to obtain the end-to-end prediction loss; determining the weights corresponding to each of the local prediction losses and the end-to-end prediction losses; and summing the weighted local prediction losses and the end-to-end prediction losses to obtain the joint prediction loss.

[0016] Optionally, updating the model parameters of the initial performance evaluation model based on the joint prediction loss includes: performing backpropagation on the joint prediction loss to obtain the model parameter gradients corresponding to the plurality of performance modeling layers and the compensation layer; wherein each of the local prediction losses participates in the calculation of the model parameter gradients of the corresponding performance modeling layer at least, and the end-to-end prediction loss participates in the calculation of the model parameter gradients of the plurality of performance modeling layers and the compensation layer; and updating the model parameters of the plurality of performance modeling layers and the compensation layer based on the model parameter gradients.

[0017] Optionally, updating the model parameters of the plurality of performance modeling layers and the compensation layer according to the model parameter gradient includes: using an optimizer to update the model parameters of the initial performance evaluation model according to the model parameter gradient; using the updated model parameters to re-obtain each of the predicted local performance evaluation results and the predicted end-to-end performance evaluation results, and redetermining the joint prediction loss; if the redetermined joint prediction loss does not meet the training stopping condition, repeatedly updating the model parameters until the joint prediction loss meets the training stopping condition.

[0018] Optionally, measuring the end-to-end performance of the target model when it completes the preset running task to obtain the measured end-to-end performance value includes: deploying the target model in the sample running environment, measuring the running time consumed by the target model from the start of the preset running task to the completion of the preset running task, obtaining the measured end-to-end latency value, and using the measured end-to-end latency value as the measured end-to-end performance value; measuring the local performance of each logic processing module when it completes the corresponding processing task to obtain the measured local performance value corresponding to each logic processing module includes: deploying the logic processing module corresponding to each performance modeling layer in the sample running environment, measuring the running time consumed by each logic processing module to complete the corresponding processing task, obtaining the measured local latency value corresponding to each logic processing module, and using each measured local latency value as the corresponding measured local performance value.

[0019] According to one aspect of this disclosure, a model performance evaluation device is provided, comprising: an acquisition module for acquiring configuration parameters of a target model and a running environment to be evaluated; a processing module for inputting the configuration parameters into a performance evaluation model, wherein the performance evaluation model includes multiple sequentially connected performance modeling layers and a compensation layer, each performance modeling layer corresponding to multiple logical processing modules executed sequentially during the operation of the target model, the multiple performance modeling layers being connected sequentially according to the execution order of the corresponding logical processing modules; the processing module is further configured to input the configuration parameters into the multiple performance modeling layers for step-by-step feature processing, wherein an intermediate feature vector output by a previous performance modeling layer is input into a subsequent performance modeling layer until an intermediate feature vector output by the final performance modeling layer is obtained; and a compensation module for inputting the intermediate feature vector output by the final performance modeling layer into the compensation layer to compensate for the cumulative error generated by the step-by-step feature processing of the multiple performance modeling layers, thereby obtaining the performance evaluation result of the target model in the running environment to be evaluated.

[0020] According to one aspect of this disclosure, a training apparatus for a performance evaluation model is provided. The initial performance evaluation model to be trained includes multiple sequentially connected performance modeling layers and compensation layers. Each performance modeling layer corresponds to multiple logical processing modules executed sequentially during the operation of the target model. The multiple performance modeling layers are connected sequentially according to the execution order of the corresponding logical processing modules. The apparatus includes: a sample acquisition module, used to acquire sample configuration parameters of the target model running in a sample running environment, end-to-end performance labels corresponding to the target model, and local performance labels corresponding to each of the logical processing modules; and a training module, used to input the sample configuration parameters into the multiple performance modeling layers for step-by-step feature processing, obtain the predicted local performance evaluation results corresponding to each performance modeling layer, and obtain the intermediate feature vector output by the final performance modeling layer. The module is further configured to input the intermediate feature vector output by the final performance modeling layer into the compensation layer to compensate for the cumulative error generated by the step-by-step feature processing of the multiple performance modeling layers, thereby obtaining the prediction end-to-end performance evaluation result; the training module is further configured to determine the corresponding local prediction loss for each performance modeling layer based on the prediction local performance evaluation result corresponding to that performance modeling layer and the local performance label of the logic processing module corresponding to that performance modeling layer; the training module is further configured to determine the end-to-end prediction loss based on the prediction end-to-end performance evaluation result and the end-to-end performance label; the training module is further configured to determine the joint prediction loss based on each local prediction loss and the end-to-end prediction loss, and update the model parameters of the initial performance evaluation model based on the joint prediction loss until the training stopping condition is met, thereby obtaining the performance evaluation model.

[0021] According to one aspect of this disclosure, an electronic device is proposed, the electronic device including a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for implementing connection communication between the processor and the memory, wherein the program is executed by the processor to implement the model running performance evaluation method or the training method of the performance evaluation model as described above.

[0022] According to one aspect of this disclosure, a computer-readable storage medium is provided, which stores one or more programs that can be executed by one or more processors to implement the model performance evaluation method or the training method for the performance evaluation model as described above.

[0023] This disclosure proposes a method for evaluating model runtime performance, a training method for the performance evaluation model, and an apparatus. The method includes: acquiring configuration parameters of a target model and the runtime environment to be evaluated; inputting the configuration parameters into a performance evaluation model, wherein the performance evaluation model includes multiple sequentially connected performance modeling layers and a compensation layer, each performance modeling layer corresponding to multiple logical processing modules executed sequentially during the operation of the target model, the multiple performance modeling layers being connected sequentially according to the execution order of the corresponding logical processing modules; inputting the configuration parameters into the multiple performance modeling layers for step-by-step feature processing, wherein the intermediate feature vector output by the previous performance modeling layer is input into the next performance modeling layer, until the intermediate feature vector output by the final performance modeling layer is obtained; inputting the intermediate feature vector output by the final performance modeling layer into the compensation layer to compensate for the cumulative error generated by the step-by-step feature processing of the multiple performance modeling layers, thereby obtaining the performance evaluation result of the target model in the runtime environment to be evaluated. This disclosure, through the structured decomposition of the internal logical processing modules of the target model and the hierarchical feature injection of configuration parameters, independently completes runtime performance evaluation even when the details of the runtime environment are uncertain or unknown. Meanwhile, the compensation layer effectively corrects the accumulated deviation in the cascaded transmission, thereby significantly improving the prediction accuracy of end-to-end performance evaluation under complex calculation processes, while taking into account both the practical feasibility of the evaluation scheme and the reliability of the results.

[0024] Furthermore, by extracting and fusing differentiated features from the common configuration parameters and the hierarchical configuration parameters of the initial performance modeling layer, the input features of the initial layer can both bear the common influence of the global operating environment and reflect the individual configuration of the initial module. This ensures that the initial feature expression of the cascaded processing is more complete and accurate, providing a high-quality data foundation for subsequent step-by-step transmission.

[0025] Furthermore, by fusing the intermediate feature vectors output by the previous performance modeling layer with the hierarchical configuration features of the current layer, the smooth transmission of upstream computing status and resource usage information can be achieved, ensuring that each subsequent layer can perceive the cumulative impact of the preceding modules, thereby accurately capturing the data dependency and resource competition effects during operation and enhancing the model's adaptability to dynamically changing environments.

[0026] Furthermore, by performing independent performance mapping on the intermediate feature vectors output by each performance modeling layer, the local performance evaluation results corresponding to each logic processing module can be obtained. This enables fine-grained performance evaluation while providing overall prediction, facilitating targeted module-level optimization by users.

[0027] Furthermore, by normalizing and feature mapping the numerical common configuration parameters, and performing discrete encoding and embedding processing on the categorical common configuration parameters, and then concatenating and fusing the two types of feature vectors, the problem of difficulty in uniformly representing heterogeneous configuration data is effectively solved, and the generalization ability and representation accuracy of global environment feature expression are improved.

[0028] Furthermore, by standardizing and embedding the numerical and categorical features in the hierarchical configuration parameters, and then concatenating them with the intermediate feature vector output from the previous layer, the integrity and fidelity of the local configuration information at the hierarchical level are ensured, and the upstream context features are effectively transmitted. This is beneficial to enhancing the discriminative power and prediction accuracy of the input features of the current performance modeling layer.

[0029] Other features and advantages of this disclosure will be set forth in the following description and will be apparent in part from the description or may be learned by practicing the disclosure. The objectives and other advantages of this disclosure may be realized and obtained by means of the structures particularly pointed out in the description, claims and drawings. Attached Figure Description

[0030] The accompanying drawings are provided to further understand the technical solutions of this disclosure and constitute a part of the specification. They are used together with the embodiments of this disclosure to explain the technical solutions of this disclosure and do not constitute a limitation on the technical solutions of this disclosure.

[0031] Figure 1 This is a system architecture diagram of the model runtime performance evaluation method applied in the embodiments of this disclosure;

[0032] Figure 2 This is a main flowchart of the model runtime performance evaluation of one embodiment of this disclosure;

[0033] Figure 3 yes Figure 2 A sub-flowchart of step S230;

[0034] Figure 4 yes Figure 2 Another sub-flowchart of step S230;

[0035] Figure 5 This is a schematic diagram of step S240 of an embodiment of the present disclosure;

[0036] Figure 6 This is a main flowchart of the training of a performance evaluation model according to an embodiment of this disclosure;

[0037] Figure 7 yes Figure 6 A sub-flowchart of step S602;

[0038] Figure 8 This is a schematic diagram of step S606 of an embodiment of the present disclosure.

[0039] Figure 9 This is a schematic diagram of the structure of a model performance evaluation device provided in one embodiment of the present disclosure;

[0040] Figure 10 This is a schematic diagram of the structure of a training device for a performance evaluation model provided in one embodiment of the present disclosure;

[0041] Figure 11 This is a schematic diagram of the structure of an electronic device proposed in one embodiment of the present disclosure. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this disclosure clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the scope of this disclosure.

[0043] Before providing a further detailed description of the embodiments of this disclosure, the terms and concepts used in these embodiments are explained, and they are subject to the following interpretations:

[0044] The runtime environment refers to the sum of the software and hardware environments upon which a program, model, or system depends for its operation. Specifically, it includes external conditions such as hardware resources, software platform, system configuration, network conditions, and data and task load, which determine whether the program or model can run and its performance. In this solution, the runtime environment can refer to the set of external conditions affecting the performance of the target model and its various logical processing modules during deployment and operation, including software and hardware conditions, system configuration, runtime parameters, and task load. This is characterized by configuration parameters. The runtime environment to be evaluated is the runtime environment in which the performance of the target model needs to be predicted, while the sample runtime environment is the runtime environment in which the target model and its various logical processing modules are actually deployed during the training of the performance evaluation model to collect end-to-end performance labels and local performance labels.

[0045] Optionally, the operating environment can be provided by artificial intelligence infrastructure (AI infrastructure), which refers to the infrastructure system that provides hardware and software resources and services for training and running artificial intelligence models.

[0046] Performance evaluation: In system design and optimization, this refers to the process of quantifying the operational efficiency of an object being evaluated through benchmarking or simulation prediction. Core evaluation metrics may include: latency, which characterizes the end-to-end time from input data to output result, typically measured in milliseconds (ms) or microseconds (μs); throughput, which characterizes the number of tasks successfully processed by the system per unit of time, typically measured in requests per second or floating-point operations per second; and resource utilization, such as video memory utilization, GPU core utilization, and memory bandwidth utilization.

[0047] Under the hierarchical evaluation system, configuration parameters are divided into different granularities, including global / public configurations, which apply to the top-level settings of the entire deployment instance, such as hardware platform model, driver version, total video memory size, and distributed communication backend. It also includes hierarchical / local configurations, which apply to fine-grained settings of specific stages or operators in the computation graph, such as the number of groups in a convolutional layer and the number of attention heads.

[0048] Feature vectors: In machine learning and signal processing, feature vectors are vectors that encode and extract raw input objects (such as configuration parameters) and transform them into fixed-length vectors. These vectors reside in a high-dimensional embedding space and are used to quantify the essential attributes of the input objects, facilitating subsequent model computations or similarity comparisons.

[0049] Vector concatenation is an operation that joins multiple independent vectors end-to-end along a specific dimension (usually the feature dimension) to form a higher-dimensional vector. This operation does not change the internal values ​​of each vector; it only performs a sequential merging at the memory or logical level. It can be used in the fusion stage of multimodal features or multi-source heterogeneous information.

[0050] Training samples: refer to instances of input data used to train the model.

[0051] Performance labels: These can refer to the actual observed values ​​corresponding to the training samples, i.e., the real performance metrics obtained through actual physical deployment and operation measurements, such as measured latency in milliseconds. Performance labels can be used to guide the optimization direction of model parameters.

[0052] Prediction loss is a core quantitative metric in the training process of machine learning models. It measures the deviation between the model's predicted output and the true label corresponding to that input. Its value directly reflects the accuracy of the model's current fit to the training samples and is the fundamental basis for guiding the iterative update of model parameters. The specific mathematical form of this metric varies depending on the task type. In regression tasks, mean squared error or mean absolute error is typically used, while cross-entropy loss is more common in classification tasks. Regardless of the form, its essence is a measure of the difference between the true and predicted values. The fundamental goal of training is to continuously reduce this loss value until convergence through algorithm optimization.

[0053] Logarithmic Mean Squared Error (Log-MSE): This is a type of mean squared error loss. It is calculated by taking the natural logarithm (or common logarithm) of both the predicted and actual values, and then calculating the squared mean of the differences between them. Since performance metrics (such as latency) typically span multiple orders of magnitude (from microseconds to seconds), logarithmic transformation can effectively reduce the difference in numerical scale, decrease the dominance of gradients by large numerical samples, and make the model more sensitive to relative errors.

[0054] Training stopping condition: refers to the criterion used to terminate the training loop in iterative optimization algorithms (such as gradient descent).

[0055] When evaluating the performance of a target model in the environment being evaluated, it is typically necessary to predict the AI ​​model's performance in that specific environment to guide selection and deployment. Related technologies mainly rely on two approaches: one is to estimate the theoretical peak performance based on the parameters of the environment being evaluated; the other is to obtain performance data through instruction-level or microarchitecture-level simulations based on the details of the underlying hardware architecture. However, both of these approaches depend on the hardware architecture details corresponding to the environment being evaluated. In real-world applications, these hardware architecture details may be in an undetermined or confidential stage, making performance evaluation difficult or resulting in low reliability of the evaluation results.

[0056] Based on this, this disclosure proposes a model runtime performance evaluation method. By decomposing the runtime chain of the target model into multiple sequentially executed logical processing modules and performing step-by-step feature processing with corresponding performance modeling layers, and setting a compensation layer at the end to correct cascaded cumulative errors, end-to-end performance evaluation can be completed based solely on available configuration parameters without needing to know the details of the underlying hardware implementation. This effectively improves the reliability of the evaluation results while ensuring the practical feasibility of the evaluation scheme.

[0057] System architecture description applied in the embodiments of this disclosure

[0058] Figure 1This is a system architecture diagram of the model performance evaluation method used in this embodiment of the disclosure, which includes: object terminal 110, Internet 120, gateway 130, and server 140.

[0059] Object terminal 110 is a device used by the object to input a performance evaluation request for model operation. It can take various forms, including desktop computers, laptops, PDAs (personal digital assistants), mobile phones, in-vehicle terminals, home theater terminals, and dedicated terminals. Furthermore, it can be a single device or a collection of multiple devices. For example, multiple devices can be connected via a local area network, sharing a single display device to work collaboratively, forming a single terminal. Object terminal 110 can also communicate with the Internet 120 via wired or wireless means to exchange data.

[0060] Gateway 130, also known as an internetwork connector or protocol converter, is a computer system or device that enables network interconnection at the transport layer and acts as a translator. It bridges the gap between two systems using different communication protocols, data formats, languages, or even completely different architectures. Gateway 130 also provides filtering and security functions. Messages sent from target terminal 110 to server 140 are forwarded to the corresponding server 140 via gateway 130. Messages sent from server 140 to target terminal 110 are also forwarded to the corresponding target terminal 110 via gateway 130.

[0061] Server 140 refers to a computer system capable of providing performance evaluation services to target terminal 110. Compared to target terminal 110, server 140 has higher requirements in terms of stability, security, and performance. Server 140 can be a single high-performance computer in a network platform, a cluster of multiple high-performance computers, a portion of a single high-performance computer (e.g., a virtual machine), or a combination of portions of multiple high-performance computers (e.g., virtual machines). Server 140 can also communicate with the Internet 120 via wired or wireless means to exchange data.

[0062] Overall Implementation Method of the Support Model Running Performance Evaluation Method in the Embodiments of this Disclosure

[0063] This disclosure provides a model performance evaluation method, applied to an apparatus supporting the model performance evaluation method, with reference to... Figure 2 The model performance evaluation method includes the following steps.

[0064] Step S210: Obtain the configuration parameters of the target model and the operating environment to be evaluated.

[0065] Step S220: Input the configured parameters into the performance evaluation model.

[0066] The performance evaluation model includes multiple performance modeling layers and compensation layers connected in sequence. Each performance modeling layer corresponds to multiple logical processing modules that are executed sequentially during the operation of the target model. The multiple performance modeling layers are connected in sequence according to the execution order of the corresponding logical processing modules.

[0067] Step S230: Input the configuration parameters into multiple performance modeling layers for step-by-step feature processing.

[0068] In this process, the intermediate feature vector output by the previous performance modeling layer is input into the next performance modeling layer, until the intermediate feature vector output by the final performance modeling layer is obtained.

[0069] Step S240: Input the intermediate feature vector output by the final performance modeling layer into the compensation layer to compensate for the cumulative error generated by the step-by-step feature processing of multiple performance modeling layers, and obtain the performance evaluation result of the target model in the operating environment to be evaluated.

[0070] In step S210, the target model can refer to the deep learning model whose performance is to be evaluated, and its structure is limited by model parameters.

[0071] Alternatively, model configuration files can be loaded from a model repository, model structure definitions can be read from training tasks, or model information can be exported from deployed services.

[0072] The operating environment to be evaluated can refer to the combination of hardware and software environments in which the target model runs. The operating environment to be evaluated can be an online cluster, an offline testbed, or a simulation environment, etc., which is related to the actual operating requirements of the target model and the operating environment.

[0073] The hardware environment can include GPU (Graphics Processing Unit) model, memory capacity, memory bandwidth, computing power of computing units, number of GPUs in a node, etc., while the software environment can include KV (Key-Value) cache data type, operator implementation library (such as cutlass), quantization strategy, etc.

[0074] Configuration parameters can refer to the various configurable variables that describe the target model when it runs in the environment to be evaluated, including common configuration parameters and hierarchical configuration parameters corresponding to each logical processing module.

[0075] Alternatively, configuration parameters can be obtained by means of user input interface, query from cluster resource management system, collection from performance testing tool or reading from preset configuration file.

[0076] The solution provided in this application requires at least the acquisition of input feature parameters required by the corresponding logical processing modules of each performance modeling layer during execution, such as sequence length, batch size, and distributed parallel strategy.

[0077] Optionally, the configuration parameters of the target model and the operating environment to be evaluated are pre-organized by the user into a structured configuration file and loaded uniformly through the configuration file reading interface.

[0078] For example, the user writes a config.json file containing structural parameters such as the target model's model type, hidden layer size (5120), and number of layers (64), as well as "configuration parameters" in the "running environment to be evaluated," including the GPU model (H200), memory bandwidth (4.8e12 BFloat16 Tensor Core computing power (9.89e14), distributed parallelism strategy (data parallelism 1, tensor parallelism 2), evaluation scene batch size (16), input sequence length (3000), and output sequence length (1000). The system reads this file through a JSON (JavaScript Object Notation) parser, performs field validation, and stores it as a configuration parameter object.

[0079] In step S220, the performance evaluation model can refer to a pre-trained deep learning model used to predict the performance of the target model in a specified operating environment.

[0080] The performance evaluation model consists of multiple sequentially connected performance modeling layers and compensation layers. The network structure of the performance evaluation model is a fully connected neural network or a Transformer structure.

[0081] The input interface definition of the performance evaluation model matches the data structure of the configuration parameters, and the output is the performance evaluation result.

[0082] Optionally, the performance evaluation model has been instantiated and loaded with training weights before performing this step, and is in a state where it can receive input data.

[0083] The performance modeling layer can refer to the network layer in the performance evaluation model used to simulate the performance behavior of a single logical processing module during the operation of the target model.

[0084] Each performance modeling layer corresponds to a logical processing module that is executed sequentially during the execution of the target model. It is used to receive the hierarchical configuration parameters of the logical processing module and the intermediate feature vectors passed from upstream, and output the intermediate feature vectors corresponding to the logical processing module after feature transformation.

[0085] The number of performance modeling layers is the same as the number of logical processing modules in the target model. Each performance modeling layer is connected sequentially according to the execution order of the corresponding logical processing modules during the operation of the target model.

[0086] The compensation layer can refer to a network layer connected to the final performance modeling layer in the performance evaluation model. It is used to receive the intermediate feature vector output by the final performance modeling layer and to compensate for the cumulative error generated by the step-by-step feature processing of multiple performance modeling layers.

[0087] Optionally, the compensation layer is not targeted at any specific logic processing module in the target model. Its function is to learn and fit the end-to-end prediction bias caused by unpredictable factors such as hardware microarchitecture residuals and dynamic scheduling overhead.

[0088] Optionally, the network structure of the compensation layer can be a fully connected layer or a Transformer layer.

[0089] A logic processing module can refer to a computational or communication unit with a clear physical or logical meaning, divided according to the execution order during the operation of the target model. For example, logic processing modules include attention calculation modules, multi-head latent attention modules, hybrid expert modules, general matrix multiplication modules, and so on. Each logic processing module is executed sequentially according to the forward propagation computation graph during the operation of the target model.

[0090] Sequential connection can refer to the output of the previous performance modeling layer in the performance evaluation model being used as part of the input of the next performance modeling layer, and being connected step by step from front to back according to the execution order of the logic processing modules to form a unidirectional data transmission path.

[0091] In one possible implementation, all acquired configuration parameters are transmitted as a unified data object to the input interface of the performance evaluation model. After receiving the unified data object, the input layer of the performance evaluation model automatically distributes the common configuration parameters in the configuration parameters to the common input channel according to the preset field mapping table, and distributes the configuration parameters of each level to the input channel of the corresponding performance modeling layer.

[0092] In step S230, the intermediate feature vector can refer to the dense vector representation output by any performance modeling layer after performing feature transformation on its input features.

[0093] Optionally, the intermediate feature vector is a fixed-length floating-point tensor in terms of dimension, which encodes the configuration information of the corresponding logical processing module of the current performance modeling layer, the scale characteristics of the input data, and the cumulative state information passed down from all upstream modules.

[0094] The intermediate feature vector is both the output of the current performance modeling layer and part of the input features of the next performance modeling layer. The dimension of the intermediate feature vector is predefined by the network structure of the performance evaluation model, and the dimension of the intermediate feature vector output by each performance modeling layer remains consistent to ensure that the subsequent performance modeling layer can correctly receive the output of the previous performance modeling layer.

[0095] The final performance modeling layer can be the last performance modeling layer in the performance evaluation model arranged according to the execution order of the corresponding logical processing modules. The input of the final performance modeling layer is the fusion result of the intermediate feature vector output by the previous performance modeling layer and its own hierarchical configuration feature vector. Its output is the final intermediate feature vector, which is then input into the compensation layer for cumulative error compensation processing.

[0096] Hierarchical feature processing refers to starting from the initial performance modeling layer and sequentially inputting the intermediate feature vectors output by the previous performance modeling layer into the next performance modeling layer, following the connection order of multiple performance modeling layers. The subsequent performance modeling layer then continues the iterative calculation of feature transformation. The initial performance modeling layer is the first performance modeling layer executed among multiple sequentially connected performance modeling layers.

[0097] For example, starting from the initial performance modeling layer, the corresponding input feature vector is passed into the initial performance modeling layer to start feature transformation. After calculating the intermediate feature vector output by this layer, the intermediate feature vector is immediately fused with the hierarchical configuration feature vector of the next performance modeling layer, and the fusion result is passed into the next performance modeling layer for feature transformation.

[0098] Furthermore, the configuration parameters include common configuration parameters and hierarchical configuration parameters corresponding to each performance modeling layer.

[0099] Please refer to Figure 3 This illustrates an exemplary embodiment of the present application, which provides a process for inputting configuration parameters into multiple performance modeling layers for hierarchical processing. This process may include the following steps:

[0100] Step S301: Perform feature processing on the common configuration parameters to obtain the common feature vector.

[0101] Among them, common configuration parameters can refer to the configuration information used to describe the global state of the operating environment to be evaluated and the overall structure of the target model. These are common parameters that affect all performance modeling layers.

[0102] For example, common configuration parameters may include hardware parameters, distributed deployment schemes, evaluation scenario parameters, and global structural parameters of the target model, etc.

[0103] A common feature vector can refer to a dense vector obtained by feature encoding all fields in a common configuration parameter.

[0104] Optionally, the common feature vector is a floating-point tensor, the dimension of which is predefined by the network structure of the performance evaluation model, and the specific value depends on the number of numeric fields, the number of categorical fields, and the embedding dimension configuration in the common configuration parameters.

[0105] The common feature vector is used as global context information input to the initial performance modeling layer in subsequent step-by-step feature processing, and can indirectly affect the calculation of other performance modeling layers through residual connections or attention mechanisms.

[0106] Feature processing can refer to a series of transformation operations that map the original configuration parameters from the input space to the feature space.

[0107] For numerical common configuration parameters, feature processing can be achieved through normalization and feature mapping; for categorical common configuration parameters, feature processing can be achieved through discrete value encoding and embedding. The feature vectors processed by the two types are then concatenated to form the final common feature vector.

[0108] Optionally, all fields in the public configuration parameters are arranged into a fixed-order field sequence according to a predefined field order, and then the values ​​of each field are read and uniformly encoded.

[0109] Further, please refer to Figure 4 The document illustrates a flowchart of a process for determining a common feature vector, as provided in an exemplary embodiment of this application, including the following steps.

[0110] Step S401: Determine the numerical common configuration parameters and the categorical common configuration parameters from the common configuration parameters.

[0111] Numerical common configuration parameters refer to configuration fields in common configuration parameters whose values ​​are continuous real numbers or discrete integers, and whose values ​​are comparable and mathematically meaningful. Classified common configuration parameters refer to configuration fields in common configuration parameters whose values ​​are a finite number of discrete labels, and whose labels do not have numerical computational meaning or ordering relationships.

[0112] Numerical common configuration parameters can directly participate in addition, subtraction, multiplication, division, and comparison operations. Their values ​​reflect quantitative differences in hardware or computing capabilities, and their ranges are typically continuous or exhibit an arithmetic progression. Numerical common configuration parameters can include hardware parameters such as memory bandwidth, memory capacity, and computing unit power; model structure parameters such as hidden layer size, number of attention heads, and number of layers; distributed deployment schemes such as data parallelism and tensor parallelism; and evaluation scenario parameters such as batch size, input sequence length, and output sequence length.

[0113] The values ​​of categorical common configuration parameters are essentially category identifiers. For example, categorical common configuration parameters include model type inference framework, KV cache data type, quantization strategy, activation function type, normalization method, interconnection method (such as NVLink, PCIe), and quantization strategy. The set of values ​​for categorical common configuration parameters is predefined before model training, and the number of parameters is typically tens to hundreds of categories.

[0114] In one implementation, a pre-defined list of common configuration parameter field types is provided, recording the name of each common configuration parameter field and its corresponding data type tag in key-value pairs. By traversing all fields in the common configuration parameters obtained in step S210, the field name is read one by one and matched against the pre-defined field type list. If the query result is a numeric tag, the field is assigned to the numeric common configuration parameter set; if the query result is a categorical tag, the field is assigned to the categorical common configuration parameter set.

[0115] In another implementation, the values ​​of each field in the public configuration parameters are read in real time, and the data type of each field is dynamically determined by probe detection. First, it is checked whether the value of the field is numeric. If so, it is directly determined to be a numeric public configuration parameter. If the value of the field is string, it is further checked whether the string can be parsed into a valid numeric value. If it can be parsed, it is a numeric public configuration parameter; otherwise, it is a variable type. If the value of the field is boolean, it is uniformly determined to be a variable type public configuration parameter.

[0116] Step S402: Normalize the numerical common configuration parameters and perform feature mapping on the normalized numerical common configuration parameters to obtain common numerical feature vectors.

[0117] Feature mapping refers to the operation of projecting a normalized numerical sequence onto a predetermined fixed-dimensional feature space through a learnable linear or nonlinear transformation layer. On the one hand, feature mapping can convert variable-length numerical sequences into fixed-dimensional vectors, enabling performance evaluation models to handle different numbers of input fields; on the other hand, it can extract cross-correlation features between fields through a learnable mapping matrix, enriching the expressive power of numerical features.

[0118] Optionally, the system independently performs maximum and minimum value normalization on each field of the numerical common configuration parameters. For each field, the original value is compressed to the [0,1] interval based on the global maximum and global minimum values ​​statistically obtained from the training dataset.

[0119] Optionally, the normalized numerical common configuration parameters are discretized into category indices, and then a Onehot mapping is performed on each category index to expand each field into a high-dimensional vector, while retaining the original normalized values ​​as continuous features. The Onehot vectors of all fields are concatenated with the continuous values ​​and then input into a fully connected linear layer to map to a predetermined dimension, outputting a common numerical feature vector.

[0120] Step S403: Perform discrete value encoding on the categorized common configuration parameters, and then embed the categorized common configuration parameters after discrete value encoding to obtain the common category feature vector.

[0121] Discrete value encoding refers to the mapping operation that converts the original category labels of each field in the categorized common configuration parameters into integer indices. Each category label is assigned a unique integer for subsequent index queries in the embedding layer. Discrete value encoding does not change the number of fields, only the representation of the values ​​in each field.

[0122] Optionally, the embedding process can map the integer indices after discrete value encoding to dense floating-point vectors through the embedding layer.

[0123] The common category feature vector can refer to a fixed-length dense floating-point vector obtained by concatenating or fusing the embedding vectors of all fields after sequentially performing discrete value encoding and embedding processing on the common configuration parameters of the classification.

[0124] In one possible implementation, each field in the categorized common configuration parameters maintains a mapping table from category to integer index, with each category assigned a unique integer starting from 0. The corresponding integer index can be obtained by looking up the original category label of each field in the table.

[0125] For example, the common configuration parameters for each type include model type (value is transformer-a), inference framework (value is framework-x), and KV cache data type (value is dtype-a). The mapping table corresponding to each field includes model type {transformer-a:0, transformer-b:1, transformer-c:2}, inference framework {framework-x:0, framework-y:1, framework-z:2}, and KV cache data type {dtype-a:0, dtype-b:1, dtype-c:2}. After discrete value encoding, the integer index [0,0,0] is obtained.

[0126] Step S404: Concatenate the common numerical feature vector and the common category feature vector to obtain the common feature vector.

[0127] A common feature vector can be a fused vector obtained by concatenating a common numerical feature vector with a common category feature vector. It is used to fully represent all the semantic information of the common configuration parameters and serves as the global context input for subsequent step-by-step feature processing.

[0128] In one possible implementation, two vectors are concatenated end to end along the channel dimension (i.e., the last dimension of the vector), with the common numerical feature vector first and the common categorical feature vector second, and the concatenation results in a common feature vector.

[0129] For example, the common numerical feature vector is 256-dimensional, and the common category feature vector is 384-dimensional. The system concatenates them along the channel dimensions to obtain a 640-dimensional common feature vector.

[0130] In another possible implementation, the dimensionality ratio of the two features is detected before concatenation. If the dimensionality difference exceeds a preset threshold (e.g., one class's dimension is more than three times that of the other), the system inputs the feature vector of the smaller class into a lightweight linear mapping layer, mapping it to the same or similar dimension as the feature vector of the other class, making the dimensionality distribution of the two classes more balanced after concatenation. After dimensional alignment, the two vectors are concatenated along the channel dimension to obtain a common feature vector.

[0131] For example, the common numerical feature vector is 128-dimensional, and the common class feature vector is 640-dimensional, with the dimensional difference exceeding a threshold. The system inputs the common numerical feature vector into a linear mapping layer (128→512 dimensions), and after mapping, it is concatenated with the 640-dimensional common class feature vector to obtain a 1152-dimensional common feature vector.

[0132] In this embodiment, by normalizing and feature mapping the numerical common configuration parameters, and performing discrete encoding and embedding processing on the categorical common configuration parameters, and then concatenating and fusing the two types of feature vectors, the problem of heterogeneous configuration data being difficult to represent uniformly is effectively solved, and the generalization ability and representation accuracy of global environment feature expression are improved.

[0133] Step S302: Perform feature processing on the hierarchical configuration parameters corresponding to the initial performance modeling layer to obtain the initial layer configuration feature vector.

[0134] Hierarchical configuration parameters refer to configuration information specifically describing a particular logical processing module, including the operator implementation type, block size, caching strategy, and fusion switch of that logical processing module. The hierarchical configuration parameters corresponding to the initial performance modeling layer are the configurable variables that the first logical processing module of the target model depends on during execution.

[0135] Step S303: Determine the input feature vector of the initial performance modeling layer based on the common feature vector and the initial layer configuration feature vector.

[0136] In one possible implementation, a common feature vector and a starting layer configuration feature vector are obtained, and both are input into their respective gating networks. Each gating network consists of a fully connected layer combined with a sigmoid activation function, outputting a gating weight vector with the same dimension as the input vector. The system multiplies the common feature vector element-wise with its gating weight vector, and multiplies the starting layer configuration feature vector element-wise with its gating weight vector. Then, the two weighted vectors are concatenated along the channel dimension to obtain the input feature vector of the starting performance modeling layer.

[0137] Step S304: Input the input feature vector of the initial performance modeling layer into the initial performance modeling layer for feature transformation to obtain the intermediate feature vector output by the initial performance modeling layer.

[0138] Feature transformation can refer to the linear or nonlinear transformation operation applied to the input feature vector by the performance modeling layer. Feature transformation maps the input vector from the original feature space to a new feature space. The dimension of the output vector can be the same as or different from the dimension of the input vector, depending on the network structure.

[0139] In one possible implementation, the initial performance modeling layer is implemented as a Transformer encoder structure, comprising a multi-head self-attention sublayer and a feedforward network sublayer, with each sublayer followed by a residual connection and layer normalization. The input feature vector of the initial performance modeling layer is used as the input to this Transformer encoder. First, the multi-head self-attention sublayer calculates the attention weights between the different parts of the input vector, and the weighted aggregation yields the attention output. The attention output is added to the original input via a residual connection, then normalized before being fed into the feedforward network sublayer. The feedforward network output, after being normalized again via a residual connection, yields the intermediate feature vector output by the initial performance modeling layer. The Transformer structure can capture the long-range dependencies between the dimensions within the input feature vector.

[0140] Step S305: For the current performance modeling layer after the initial performance modeling layer, determine the layer configuration parameters corresponding to the current performance modeling layer from the configuration parameters.

[0141] In one possible implementation, a unique layer index number is pre-assigned to each performance modeling layer, and a mapping table is established between the layer index number and the name of the corresponding logical processing module for each performance modeling layer.

[0142] For example, the predefined mapping table includes: Layer Index 1 → Attention Module, Layer Index 2 → Expert Parallel Module, and Layer Index 3 → Communication Protocol Module. When the layer index of the current performance modeling layer is 2, the mapping table is queried to obtain the corresponding logical processing module name as Expert Parallel Module. Finally, the set of layer configuration parameters with the key name "Expert Parallel Module" is searched from the configuration parameters and determined as the layer configuration parameter corresponding to the current performance modeling layer.

[0143] In another possible implementation, the hierarchical configuration parameter sets corresponding to each logical processing module in the configuration parameters are arranged into an ordered list according to the execution order of the logical processing modules during the target model's operation. The hierarchical configuration parameters corresponding to the starting performance modeling layer are placed at the beginning of the list, and the hierarchical configuration parameters corresponding to each performance modeling layer after the starting performance modeling layer are arranged sequentially according to the execution order. Using this sequence number as an index, the hierarchical configuration parameter set at the corresponding position is retrieved from the ordered list as the hierarchical configuration parameter corresponding to the current performance modeling layer.

[0144] Step S306: Perform feature processing on the layer configuration parameters corresponding to the current performance modeling layer to obtain the current layer configuration feature vector.

[0145] Further, numerical hierarchical configuration parameters and categorical hierarchical configuration parameters are determined from the hierarchical configuration parameters; the numerical hierarchical configuration parameters are normalized, and feature mapping is performed on the normalized numerical hierarchical configuration parameters to obtain hierarchical numerical feature vectors; the categorical hierarchical configuration parameters are discretely encoded, and embedding processing is performed on the discretely encoded categorical hierarchical configuration parameters to obtain hierarchical category feature vectors; the hierarchical numerical feature vectors and hierarchical category feature vectors are concatenated to obtain the current layer configuration feature vector; the intermediate feature vector output from the previous performance modeling layer and the current layer configuration feature vector are concatenated to obtain the input feature vector of the current performance modeling layer.

[0146] Step S307: Determine the input feature vector of the current performance modeling layer based on the intermediate feature vector output by the previous performance modeling layer and the configuration feature vector of the current layer.

[0147] Optionally, the integrated vector obtained by fusing the intermediate feature vector output by the previous performance modeling layer with the current layer's configured feature vector can be used as the input for the current performance modeling layer to perform feature transformation.

[0148] In one possible implementation, the intermediate feature vector output by the previous performance modeling layer is used as the query vector, and the current layer configuration feature vector is used as the key vector and value vector. Cross-attention weights are calculated: the query vector and the key vector are multiplied by a dot product to calculate a similarity score, which is then normalized by Softmax and weighted and summed with the value vector to obtain the attention fusion result.

[0149] The attention fusion result is added to the previous intermediate feature vector via residual connection, and then normalized by the layer to serve as the input feature vector for the current performance modeling layer.

[0150] Step S308: Input the input feature vector of the current performance modeling layer into the current performance modeling layer for feature transformation to obtain the intermediate feature vector output by the current performance modeling layer.

[0151] The specific implementation method of this step can be referred to step S304 above, and will not be repeated here.

[0152] Step S309: Perform the above processing on each performance modeling layer after the initial performance modeling layer in sequence until the intermediate feature vector output by the final performance modeling layer is obtained.

[0153] In this embodiment, by performing independent performance mapping on the intermediate feature vectors output by each performance modeling layer, the local performance evaluation results corresponding to each logic processing module can be obtained. This enables fine-grained performance evaluation while providing overall prediction, making it easier for users to perform targeted module-level optimization.

[0154] Furthermore, during the step-by-step feature processing, the intermediate feature vectors output by each performance modeling layer can be mapped to obtain the local performance evaluation results corresponding to each performance modeling layer.

[0155] The local performance evaluation results represent the operating performance of the corresponding logic processing module in the operating environment to be evaluated.

[0156] Optionally, local performance evaluation results may include execution latency and computational throughput.

[0157] In one possible implementation, each performance modeling layer is configured with an independent performance mapping layer, which can be implemented as a single-layer fully connected network. After the current performance modeling layer completes feature transformation and outputs an intermediate feature vector, the intermediate feature vector is simultaneously input into the independent performance mapping layer corresponding to that layer. The vector is directly mapped to a scalar value through matrix multiplication, serving as the local performance evaluation result for that performance modeling layer.

[0158] In step S240, the compensation layer can refer to a network layer connected after the final performance modeling layer in the performance evaluation model. It is used to receive the intermediate feature vector output by the final performance modeling layer and to compensate for the cumulative error generated by the step-by-step feature processing of multiple performance modeling layers.

[0159] Cumulative error can refer to the total deviation formed by the accumulation of approximation errors generated by feature transformations at each level during the hierarchical feature processing of multiple performance modeling layers along the cascaded structure.

[0160] In one possible implementation, the compensation layer is implemented as a multi-layer fully connected network, containing at least one hidden layer and a corresponding activation function, and finally a linear mapping layer.

[0161] Furthermore, the compensation feature vector output by the compensation layer is subjected to performance mapping to obtain the end-to-end performance evaluation result, and the end-to-end performance evaluation result is used as the performance evaluation result.

[0162] The end-to-end performance evaluation results include at least one of runtime latency and throughput. End-to-end performance evaluation results can refer to performance index values ​​with clear physical dimensions obtained after performance mapping of the compensation feature vector, used to characterize the overall runtime performance of the target model in the environment to be evaluated.

[0163] The intermediate feature vector output by the final performance modeling layer is input into the compensation layer. After being transformed by each fully connected layer or Transformer layer inside the compensation layer, the compensation feature vector output by the compensation layer is finally obtained.

[0164] In one exemplary implementation, the performance modeling layer can be a fully connected neural network or a Transformer network. The input feature vector is input into the fully connected neural network or Transformer network for feature transformation, resulting in an intermediate feature vector output by the performance modeling layer.

[0165] In the process of compensating for cumulative error, the intermediate feature vector output by the final performance modeling layer can be input into a fully connected neural network or a Transformer network for compensation feature extraction to obtain a compensation feature vector. Then, the compensation feature vector is mapped to the performance to obtain the performance evaluation result.

[0166] Please refer to Figure 5 For the common configuration parameter Input0, the numerical common configuration parameters are normalized and then linearly mapped to obtain a common numerical feature vector; the categorical common configuration parameters are embedded to obtain a common category feature vector. The common category feature vector and the common numerical feature vector are concatenated at the concat point to obtain a common feature vector, which is injected as global context information into each performance modeling layer.

[0167] For each performance modeling layer, the corresponding layer configuration parameter Input_n is processed as follows: the numerical layer configuration parameters are normalized and then linearly mapped to obtain the layer numerical feature vector; the categorical layer configuration parameters are embedded to obtain the layer category feature vector. The layer category feature vector and the layer numerical feature vector are concatenated at the concat point to obtain the current layer configuration feature vector, which is then input into the performance modeling layer.

[0168] Each performance modeling layer includes a transformer_n feature transformation network and an MLP output network. Each performance modeling layer takes the received current layer configuration feature vector and the intermediate feature vector output from the previous performance modeling layer as inputs, maps them through the transformer_n feature transformation network to an intermediate feature vector, and then maps the intermediate feature vector through the MLP output network to the local performance evaluation result Output_n. The figure also shows a compensation layer. The transformer_n+1 feature transformation network of the compensation layer receives the intermediate feature vector output from the previous layer, and the MLP output network of the compensation layer maps the transformed features to the end-to-end performance evaluation result Output0.

[0169] In one possible implementation, the compensation feature vector is input into the linear mapping layer, and matrix multiplication is performed to obtain an output scalar or vector. If the output is a scalar, it is directly used as the runtime latency evaluation result; if the output is a two-dimensional vector, it is used as the runtime latency evaluation result and the throughput evaluation result, respectively.

[0170] This disclosure achieves independent performance evaluation even when runtime environment details are uncertain or unknown by structurally decomposing the internal logic processing modules of the target model and injecting hierarchical features into configuration parameters. Simultaneously, a compensation layer effectively corrects accumulated deviations in cascading transmission, significantly improving the prediction accuracy of end-to-end performance evaluation under complex computational processes, while balancing the practical feasibility and reliability of the evaluation scheme.

[0171] Furthermore, by extracting and fusing differentiated features from the common configuration parameters and the hierarchical configuration parameters of the initial performance modeling layer, the input features of the initial layer can both bear the common influence of the global operating environment and reflect the individual configuration of the initial module. This ensures that the initial feature expression of the cascaded processing is more complete and accurate, providing a high-quality data foundation for subsequent step-by-step transmission.

[0172] Furthermore, by fusing the intermediate feature vectors output by the previous performance modeling layer with the hierarchical configuration features of the current layer, the smooth transmission of upstream computing status and resource usage information can be achieved, ensuring that each subsequent layer can perceive the cumulative impact of the preceding modules, thereby accurately capturing the data dependency and resource competition effects during operation and enhancing the model's adaptability to dynamically changing environments.

[0173] Furthermore, by performing independent performance mapping on the intermediate feature vectors output by each performance modeling layer, the local performance evaluation results corresponding to each logic processing module can be obtained. This enables fine-grained performance evaluation while providing overall prediction, facilitating targeted module-level optimization by users.

[0174] Furthermore, by normalizing and feature mapping the numerical common configuration parameters, and performing discrete encoding and embedding processing on the categorical common configuration parameters, and then concatenating and fusing the two types of feature vectors, the problem of difficulty in uniformly representing heterogeneous configuration data is effectively solved, and the generalization ability and representation accuracy of global environment feature expression are improved.

[0175] Furthermore, by standardizing and embedding the numerical and categorical features in the hierarchical configuration parameters, and then concatenating them with the intermediate feature vector output from the previous layer, the integrity and fidelity of the local configuration information at the hierarchical level are ensured, and the upstream context features are effectively transmitted. This is beneficial to enhancing the discriminative power and prediction accuracy of the input features of the current performance modeling layer.

[0176] In some embodiments, the performance evaluation model needs to be trained before it is adopted, so that it can perform performance evaluation on the runtime environment to be evaluated based on the configuration parameters of the runtime environment to be evaluated.

[0177] The initial performance evaluation model to be trained includes multiple performance modeling layers and compensation layers connected in sequence. Each performance modeling layer corresponds to multiple logical processing modules executed sequentially during the operation of the target model. The multiple performance modeling layers are connected in sequence according to the execution order of the corresponding logical processing modules.

[0178] Please refer to Figure 6 The diagram illustrates a flowchart of a training method for a performance evaluation model provided in an exemplary embodiment of this application, the process including the following steps.

[0179] Step S601: Obtain the sample configuration parameters of the target model running in the sample running environment, the end-to-end performance label of the target model, and the local performance label of each logic processing module.

[0180] Step S602: Input the sample configuration parameters into multiple performance modeling layers for step-by-step feature processing, obtain the prediction local performance evaluation results corresponding to each performance modeling layer, and obtain the intermediate feature vector output by the final performance modeling layer.

[0181] Step S603: Input the intermediate feature vector output by the final performance modeling layer into the compensation layer to compensate for the cumulative error generated by the step-by-step feature processing of multiple performance modeling layers, and obtain the prediction end-to-end performance evaluation result.

[0182] Step S604: For each performance modeling layer, determine the corresponding local prediction loss based on the prediction local performance evaluation result corresponding to the performance modeling layer and the local performance label of the logic processing module corresponding to the performance modeling layer.

[0183] Step S605: Determine the end-to-end prediction loss based on the predicted end-to-end performance evaluation results and the end-to-end performance label.

[0184] Step S606: Determine the joint prediction loss based on the local prediction loss and the end-to-end prediction loss, and update the model parameters of the initial performance evaluation model based on the joint prediction loss until the training stopping condition is met, thus obtaining the performance evaluation model.

[0185] In step S601, the sample running environment can refer to the specific hardware and software environment used to collect training data, and its configuration is consistent with or similar to the actual deployment scenario of the target model.

[0186] Sample configuration parameters can refer to a set of specific configuration parameter values ​​collected in the sample running environment, which serve as input features for training samples.

[0187] End-to-end performance labels can refer to the measured end-to-end performance index values ​​obtained when the target model completes the preset running tasks in the sample running environment, which can be used as the true values ​​for calculating end-to-end prediction loss during training.

[0188] Local performance labels can refer to the measured local performance index values ​​obtained by each logic processing module when it independently completes its corresponding processing task in the sample running environment, which can be used as the true values ​​for calculating the local prediction loss during training.

[0189] In one possible implementation, firstly, the target model is deployed and run in the sample running environment, the end-to-end performance of the target model when it completes the preset running task is measured, the measured end-to-end performance value is obtained, and the end-to-end performance label is determined based on the measured end-to-end performance value.

[0190] Secondly, the logic processing modules corresponding to each performance modeling layer are deployed and run in the sample runtime environment. The local performance of each logic processing module when it completes the corresponding processing task is measured, and the measured local performance values ​​of each logic processing module are obtained.

[0191] Finally, based on the measured values ​​of each local performance, the local performance labels corresponding to each logic processing module are determined.

[0192] The preset running task can refer to a complete inference or training process, such as a forward propagation plus sampling generation, with a clear start and end time.

[0193] End-to-end performance measurements can refer to the runtime or throughput of the target model in the sample operating environment when it completes the preset running tasks, which is the raw measurement data.

[0194] The corresponding processing task can refer to the part of the computational task undertaken by each logical processing module during the operation of the target model, such as the task of the attention module performing self-attention computation.

[0195] Local performance measured values ​​can refer to the running time consumed by each logic processing module to independently complete its corresponding processing task in the sample operating environment, which is the raw measurement data.

[0196] Specifically, firstly, the target model is fully deployed in the sample runtime environment, pre-configured sample parameters are set, and pre-defined tasks are run. A high-precision timer records the time elapsed from task start to finish, obtaining end-to-end performance measurements. These measurements are then used directly as end-to-end performance labels. Next, following the execution order of multiple logical processing modules during the target model's operation, each logical processing module is extracted from the target model and deployed separately in the same sample runtime environment. Corresponding sample-level configuration parameters are set, and the processing tasks for each module are run. The time elapsed for each module is measured, obtaining local performance measurements. Finally, each local performance measurement is used as a local performance label for its corresponding logical processing module.

[0197] Optionally, all measurements are performed under the same sample configuration parameters to ensure a one-to-one correspondence between labels and input features.

[0198] For example, in an H200 environment with batch_size=16 and input_len=3000, running the target model for one forward propagation yielded an end-to-end latency of 15.7ms, which was used as the end-to-end performance label. Subsequently, the attention module was deployed separately, and a latency of 3.2ms was measured as the local performance label for that module; the expert parallel module was deployed separately, and a latency of 4.1ms was measured as the local performance label for that module.

[0199] Furthermore, the target model is deployed and run in the sample runtime environment. The runtime consumed by the target model from the start of the preset running task to the completion of the preset running task is measured to obtain the end-to-end latency measurement value, which is used as the end-to-end performance measurement value. Additionally, the logic processing modules corresponding to each performance modeling layer are deployed and run in the sample runtime environment. The runtime consumed by each logic processing module to complete its corresponding processing task is measured to obtain the local latency measurement value for each logic processing module, which is used as the corresponding local performance measurement value.

[0200] Optionally, the running time consumed by each logic processing module to complete the corresponding processing task is measured to obtain the local latency measurement value corresponding to each logic processing module, and the corresponding local performance label is determined according to each local latency measurement value.

[0201] In one implementation, a start timestamp is recorded before the task begins and an end timestamp is recorded after the task is completed. The difference between the two timestamps is calculated to obtain the measured end-to-end delay value, which is then used directly as the end-to-end performance label.

[0202] In step S602, please refer to Figure 7 The diagram illustrates a flowchart of a step-by-step feature processing procedure using multi-sample configuration parameters provided in an exemplary embodiment of this application. The procedure includes the following steps.

[0203] Step S701: Determine the common configuration parameters of the samples and the sample hierarchical configuration parameters corresponding to each performance modeling layer from the sample configuration parameters.

[0204] Step S702: Perform feature processing on the common configuration parameters of the samples to obtain the common feature vector of the samples.

[0205] Step S703: Perform feature processing on the sample layer configuration parameters corresponding to the initial performance modeling layer to obtain the sample initial layer configuration feature vector.

[0206] Step S704: Determine the sample input feature vector of the initial performance modeling layer based on the sample common feature vector and the sample initial layer configuration feature vector.

[0207] Step S705: Input the sample input feature vector into the initial performance modeling layer for feature transformation to obtain the intermediate feature vector output by the initial performance modeling layer.

[0208] Step S706: For the current performance modeling layer after the initial performance modeling layer, perform feature processing on the sample layer configuration parameters corresponding to the current performance modeling layer to obtain the current layer sample configuration feature vector.

[0209] Step S707: Determine the sample input feature vector of the current performance modeling layer based on the intermediate feature vector output by the previous performance modeling layer and the sample configuration feature vector of the current layer.

[0210] Step S708: Input the sample input feature vector of the current performance modeling layer into the current performance modeling layer for feature transformation to obtain the intermediate feature vector output by the current performance modeling layer, and perform the above feature processing in sequence until the intermediate feature vector output by the final performance modeling layer is obtained.

[0211] The specific implementation methods of each step in steps S701 to S708 above can be referred to the implementation methods of the corresponding steps mentioned above, and will not be repeated here.

[0212] Furthermore, performance mapping is performed on the intermediate feature vectors output by each performance modeling layer to obtain the prediction local performance evaluation results corresponding to each performance modeling layer.

[0213] The implementation steps for performing performance mapping to predict local performance evaluation results can refer to the implementation process of performing performance mapping on the compensation feature vector output by the compensation layer in the above steps, and will not be repeated here in this embodiment.

[0214] In step S606, the local prediction losses and the end-to-end prediction losses are integrated into a joint prediction loss, which drives model convergence through backpropagation and parameter updates. The joint prediction loss ensures that each performance modeling layer is simultaneously supervised by both local physical constraints and end-to-end global constraints, guaranteeing that the model parameters are optimized in a direction that conforms to physical laws and approximates the end-to-end measured values.

[0215] The joint prediction loss can refer to the total loss scalar obtained by weighted summation of the local prediction losses and the end-to-end prediction loss, which serves as the sole source of gradients for backpropagation.

[0216] Training termination conditions can refer to preset termination criteria, including at least one of the following: joint prediction loss is lower than a preset threshold, the maximum number of training rounds is reached, or the validation set performance does not improve for several consecutive rounds.

[0217] The model parameters of the initial performance evaluation model can refer to all learnable parameters of multiple performance modeling layers and compensation layers in the performance evaluation model to be trained, including weights, biases, batch normalization parameters, and embedding layer weights.

[0218] In one possible implementation, the logarithmic mean square error between each predicted local performance evaluation result and the corresponding local performance label is first calculated to obtain the local prediction loss.

[0219] Please refer to Figure 8The initial performance evaluation model consists of a common input part (Layer 0 in the diagram), multiple sequentially connected performance modeling layers (Layer 1 to Layer _n in the diagram), and a compensation layer (Layer _n+1 in the diagram). During training, the common input part receives common configuration parameters of the samples (Input 0 in the diagram), extracts features from the common configuration parameters, and outputs a common feature vector of the samples. The multiple performance modeling layers include the initial performance modeling layer (Layer 1 in the diagram) and the subsequent performance modeling layers (Layer 2 to Layer _n in the diagram).

[0220] The initial performance modeling layer receives the common feature vector of the samples and the sample layer configuration parameters (i.e., Input1 in the diagram) as input. After forward computation, it outputs the intermediate feature vector and the predicted local performance evaluation result of the initial performance modeling layer. Based on the predicted local performance evaluation result and the corresponding local performance label, the local prediction loss of the initial performance modeling layer is calculated. For the performance modeling layers after the initial performance modeling layer, taking Layer2 in the diagram as an example, it receives the intermediate feature vector output by the previous performance modeling layer (i.e., the initial performance modeling layer Layer1) and the sample layer configuration parameters (Input2) of the current performance modeling layer (Layer2) as input. After forward computation, it outputs the intermediate feature vector and the predicted local performance evaluation result of the current performance modeling layer. Based on the predicted local performance evaluation result and the corresponding local performance label, the local prediction loss of the current performance modeling layer is calculated. Subsequent performance modeling layers (Layer3 to Layer_n) follow the same pattern. Each performance modeling layer outputs its own corresponding intermediate feature vector and predicted local performance evaluation result, and calculates its own corresponding local prediction loss.

[0221] In this process, the intermediate feature vector output by the last performance modeling layer (Layer_n) is further passed to the compensation layer (Layer_n+1). The compensation layer outputs the predicted end-to-end performance evaluation result based on this intermediate feature vector, and calculates the end-to-end prediction loss based on this predicted end-to-end performance evaluation result and the end-to-end performance label. Thus, each performance modeling layer in the multiple performance modeling layers generates its corresponding local prediction loss, the compensation layer generates the end-to-end prediction loss, and the weighted sum of each local prediction loss and the end-to-end prediction loss is used to update the parameters of the initial performance evaluation model.

[0222] Furthermore, the log-mean square error between the predicted end-to-end performance evaluation results and the end-to-end performance labels is calculated to obtain the end-to-end prediction loss.

[0223] Subsequently, the weights corresponding to each local prediction loss and the end-to-end prediction loss are determined.

[0224] Finally, the weighted local prediction losses and end-to-end prediction losses are summed to obtain the joint prediction loss.

[0225] The local prediction loss can be a scalar obtained by calculating the logarithmic mean square error between the predicted local performance evaluation results and the corresponding local performance labels, which measures the prediction bias of the performance modeling layer.

[0226] The predicted end-to-end performance evaluation result can be the end-to-end performance prediction value obtained by performance mapping from the output of the compensation layer.

[0227] End-to-end prediction loss can be a scalar obtained by calculating the logarithmic mean square error between the predicted end-to-end performance evaluation results and the end-to-end performance labels, and measures the end-to-end prediction deviation of the entire network.

[0228] For example, during model training, the log-mean-square error between the predicted local performance evaluation result and the corresponding local performance label of each performance modeling layer is calculated sequentially according to the forward propagation process, resulting in the local prediction loss Loss_i = logMSE(Y_i, Y_itrue), where Y_i is the predicted local performance evaluation result corresponding to the i-th performance modeling layer, and Y_itrue is the local performance label of the corresponding logic processing module. Simultaneously, the system calculates the log-mean-square error between the predicted end-to-end performance evaluation result and the end-to-end performance label, resulting in the end-to-end prediction loss Loss_0 = logMSE(Y_0, Y_0true), where Y_0 is the predicted end-to-end performance evaluation result output by the compensation layer, and Y_0true is the end-to-end performance label.

[0229] Subsequently, corresponding weight coefficients, denoted as α1, α2, ..., αn, α0, are assigned to each local prediction loss and the end-to-end prediction loss, respectively. Each weight is a positive real number, used to adjust the contribution ratio of prediction accuracy at different granularities to the total loss. The system multiplies each local prediction loss by its corresponding weight, and the end-to-end prediction loss by its corresponding weight, summing all weighted loss terms to obtain the joint prediction loss: Losstotal = α1·Loss1 + α2·Loss2 + ... + αn·Lossn + α0·Loss0. This joint prediction loss drives the optimization and updating of all trainable parameters of the initial performance evaluation model towards a direction that balances end-to-end optimality and local physical consistency.

[0230] Furthermore, backpropagation is performed on the joint prediction loss to obtain the model parameter gradients for multiple performance modeling layers and compensation layers. Subsequently, the model parameters of multiple performance modeling layers and compensation layers are updated based on the model parameter gradients.

[0231] In this context, each local prediction loss participates in the gradient calculation of the model parameters of the corresponding performance modeling layer at least, while the end-to-end prediction loss participates in the gradient calculation of the model parameters of multiple performance modeling layers and compensation layers.

[0232] Backpropagation can refer to the process of calculating the gradient of the loss with respect to each trainable parameter, starting from the joint prediction loss and working backward layer by layer according to the chain rule.

[0233] The gradient of the model parameters is used to characterize the partial derivative of the joint prediction loss with respect to the weight matrix and bias vector of each network layer. Its direction indicates the optimal adjustment direction for the parameters to reduce the joint prediction loss, and its magnitude indicates the adjustment step size.

[0234] Optionally, an optimizer is used to update the model parameters of the initial performance evaluation model based on the model parameter gradient. The updated model parameters are then used to re-obtain the local performance evaluation results and the end-to-end performance evaluation results, and the joint prediction loss is redefined. If the redefined joint prediction loss does not meet the training stopping criterion, the model parameters are updated repeatedly until the joint prediction loss meets the training stopping criterion.

[0235] In one possible implementation, a dual stopping condition is employed: a fixed maximum number of iterations and a loss threshold. A preset maximum number of training iterations (e.g., 200 iterations) and a joint prediction loss threshold (e.g., 0.01) are used.

[0236] For example, the system presets a maximum of 100 training epochs and a joint loss threshold of 0.005. The joint prediction loss is recalculated after each epoch's parameter update. In the 72nd epoch, the joint loss drops to 0.0048, below the threshold, and training terminates prematurely.

[0237] Optionally, the local prediction loss is not included in the gradient calculation of non-corresponding layers to avoid local supervision signals misleading non-corresponding modules.

[0238] In this embodiment, by jointly optimizing the local prediction loss and the end-to-end prediction loss, each performance modeling layer is subject to physical operator supervision and constraints while the entire network converges to the end-to-end optimal convergence. This balances hierarchical interpretability and global prediction accuracy, effectively alleviating the problems of overfitting and distortion of intermediate layer features in model regression.

[0239] Furthermore, by independently deploying and testing the target model and each logical processing module, end-to-end and local dual-granularity performance labels are obtained, providing a real and reliable source of supervision signals for joint training and supporting the computational needs of multi-granularity loss functions.

[0240] Furthermore, the specific process of feature processing at each level during the training phase is defined, so that the common configuration of samples and the hierarchical configuration are decoupled and encoded, and the upstream state is passed step by step along the cascade structure. This ensures that the data path is aligned between the training and inference phases, so that the learned parameters can be directly applied to the inference scenario.

[0241] Furthermore, the logarithmic mean square error is used to calculate each loss term and the weighted loss is summed to alleviate the sample imbalance problem when the performance index is distributed across orders of magnitude. At the same time, it supports the flexible configuration of prediction accuracy weights at each granularity, balancing local accuracy and global accuracy.

[0242] Furthermore, the gradient of each local prediction loss is applied to the corresponding performance modeling layer, and the gradient of the end-to-end prediction loss is applied to all layers and the compensation layer. The dual supervision signals work together to optimize the parameters of each layer, driving the network to converge to both physically reasonable and end-to-end optimal directions.

[0243] Furthermore, through iterative training loop, the model parameters are gradually optimized until convergence under the continuous drive of the joint prediction loss, ensuring that the final model achieves the prediction accuracy required for the application.

[0244] Furthermore, the measured running time of the target model and each module is directly used as the performance label, so that the label is strictly aligned with the dimension of the prediction target, providing a reliable supervision signal for loss calculation.

[0245] Description of apparatus and devices according to embodiments of this disclosure

[0246] See Figure 9 This disclosure also provides a model performance evaluation device 900, including an acquisition module 910, a processing module 920, and a compensation module 930.

[0247] The acquisition module 910 is used to acquire the configuration parameters of the target model and the operating environment to be evaluated;

[0248] The processing module 920 is used to input the configuration parameters into the performance evaluation model, wherein the performance evaluation model includes multiple performance modeling layers and compensation layers connected in sequence, each performance modeling layer corresponds to multiple logical processing modules executed in sequence during the operation of the target model, and the multiple performance modeling layers are connected in sequence according to the execution order of the corresponding logical processing modules.

[0249] The processing module 920 is further configured to input the configuration parameters into the plurality of performance modeling layers for step-by-step feature processing, wherein the intermediate feature vector output by the previous performance modeling layer is input into the next performance modeling layer until the intermediate feature vector output by the final performance modeling layer is obtained.

[0250] The compensation module 930 is used to input the intermediate feature vector output by the final performance modeling layer into the compensation layer to compensate for the cumulative error generated by the step-by-step feature processing of the multiple performance modeling layers, so as to obtain the performance evaluation result of the target model in the operating environment to be evaluated.

[0251] Optionally, the configuration parameters include common configuration parameters and hierarchical configuration parameters corresponding to each of the performance modeling layers;

[0252] The processing module 920 is used to perform feature processing on the common configuration parameters to obtain a common feature vector; perform feature processing on the hierarchical configuration parameters corresponding to the initial performance modeling layer to obtain an initial layer configuration feature vector; determine the input feature vector of the initial performance modeling layer based on the common feature vector and the initial layer configuration feature vector; and input the input feature vector of the initial performance modeling layer into the initial performance modeling layer for feature transformation to obtain the intermediate feature vector output by the initial performance modeling layer.

[0253] Optionally, the processing module 920 is further configured to: determine, for the current performance modeling layer following the initial performance modeling layer, a hierarchical configuration parameter corresponding to the current performance modeling layer from the configuration parameters; perform feature processing on the hierarchical configuration parameter corresponding to the current performance modeling layer to obtain a current layer configuration feature vector; determine the input feature vector of the current performance modeling layer based on the intermediate feature vector output by the previous performance modeling layer and the current layer configuration feature vector; input the input feature vector of the current performance modeling layer into the current performance modeling layer for feature transformation to obtain an intermediate feature vector output by the current performance modeling layer; and sequentially perform the above processing on each performance modeling layer following the initial performance modeling layer until the intermediate feature vector output by the final performance modeling layer is obtained.

[0254] Optionally, the processing module 920 is further configured to perform performance mapping on the intermediate feature vectors output by each performance modeling layer to obtain the local performance evaluation results corresponding to each performance modeling layer; wherein each local performance evaluation result represents the operating performance of the corresponding logic processing module in the operating environment to be evaluated.

[0255] Optionally, the processing module 920 is configured to determine numerical common configuration parameters and categorical common configuration parameters from the common configuration parameters; normalize the numerical common configuration parameters and perform feature mapping on the normalized numerical common configuration parameters to obtain a common numerical feature vector; perform discrete value encoding on the categorical common configuration parameters and perform embedding processing on the discrete value encoded categorical common configuration parameters to obtain a common category feature vector; and concatenate the common numerical feature vector and the common category feature vector to obtain the common feature vector.

[0256] Optionally, the processing module 920 is further configured to: determine numerical hierarchical configuration parameters and categorical hierarchical configuration parameters from the hierarchical configuration parameters; normalize the numerical hierarchical configuration parameters and perform feature mapping on the normalized numerical hierarchical configuration parameters to obtain hierarchical numerical feature vectors; perform discrete value encoding on the categorical hierarchical configuration parameters and perform embedding processing on the discrete value encoded categorical hierarchical configuration parameters to obtain hierarchical category feature vectors; concatenate the hierarchical numerical feature vectors and the hierarchical category feature vectors to obtain the current layer configuration feature vector; and concatenate the intermediate feature vector output by the previous performance modeling layer and the current layer configuration feature vector to obtain the input feature vector of the current performance modeling layer.

[0257] The support model performance evaluation device 900 disclosed herein is used to execute the support model performance evaluation method as described in the above embodiments. Its specific processing procedure is the same as that of the model performance evaluation method described in the above embodiments, and will not be repeated here.

[0258] See Figure 10 This disclosure also provides a training device 1000 for a performance evaluation model, including a sample acquisition module 1010 and a training module 1020. The initial performance evaluation model to be trained includes multiple sequentially connected performance modeling layers and compensation layers. Each performance modeling layer corresponds to multiple logical processing modules executed sequentially during the operation of the target model. The multiple performance modeling layers are connected sequentially according to the execution order of the corresponding logical processing modules.

[0259] The sample acquisition module 1010 is used to acquire the sample configuration parameters of the target model running in the sample running environment, the end-to-end performance label of the target model, and the local performance label of each of the logic processing modules.

[0260] The training module 1020 is used to input the sample configuration parameters into the multiple performance modeling layers for step-by-step feature processing, obtain the prediction local performance evaluation results corresponding to each performance modeling layer, and obtain the intermediate feature vector output by the final performance modeling layer.

[0261] The training module 1020 is also used to input the intermediate feature vector output by the final performance modeling layer into the compensation layer to compensate for the cumulative error generated by the step-by-step feature processing of the multiple performance modeling layers, and to obtain the prediction end-to-end performance evaluation result.

[0262] The training module 1020 is further configured to determine the corresponding local prediction loss for each performance modeling layer based on the prediction local performance evaluation result corresponding to the performance modeling layer and the local performance label of the logic processing module corresponding to the performance modeling layer.

[0263] The training module 1020 is further configured to determine the end-to-end prediction loss based on the prediction end-to-end performance evaluation result and the end-to-end performance label.

[0264] The training module 1020 is further configured to determine a joint prediction loss based on each of the local prediction losses and the end-to-end prediction losses, and update the model parameters of the initial performance evaluation model based on the joint prediction loss until the training stopping condition is met, thereby obtaining the performance evaluation model.

[0265] Optionally, the sample acquisition module 1010 is used to deploy the target model in the sample running environment, measure the end-to-end performance of the target model when it completes a preset running task, and obtain the measured end-to-end performance value; determine the end-to-end performance label based on the measured end-to-end performance value; deploy the logic processing module corresponding to each performance modeling layer in the sample running environment, measure the local performance of each logic processing module when it completes the corresponding processing task, and obtain the measured local performance value corresponding to each logic processing module; and determine the local performance label corresponding to each logic processing module based on the measured local performance value.

[0266] Optionally, the training module 1020 is configured to: determine common sample configuration parameters and sample hierarchical configuration parameters corresponding to each performance modeling layer from the sample configuration parameters; perform feature processing on the common sample configuration parameters to obtain a common sample feature vector; perform feature processing on the sample hierarchical configuration parameters corresponding to the initial performance modeling layer to obtain a sample initial layer configuration feature vector; determine the sample input feature vector of the initial performance modeling layer based on the common sample feature vector and the sample initial layer configuration feature vector; and input the sample input feature vector into the initial performance modeling layer for feature transformation to obtain the input feature vector of the initial performance modeling layer. The intermediate feature vector is generated; for the current performance modeling layer after the initial performance modeling layer, feature processing is performed on the sample layer configuration parameters corresponding to the current performance modeling layer to obtain the current layer sample configuration feature vector; based on the intermediate feature vector output by the previous performance modeling layer and the current layer sample configuration feature vector, the sample input feature vector of the current performance modeling layer is determined; the sample input feature vector of the current performance modeling layer is input into the current performance modeling layer for feature transformation to obtain the intermediate feature vector output by the current performance modeling layer, and the above feature processing is performed sequentially until the intermediate feature vector output by the final performance modeling layer is obtained.

[0267] Optionally, the training module 1020 is configured to calculate the log-mean square error between each predicted local performance evaluation result and the corresponding local performance label to obtain each local prediction loss; calculate the log-mean square error between the predicted end-to-end performance evaluation result and the end-to-end performance label to obtain the end-to-end prediction loss; determine the weights corresponding to each local prediction loss and the end-to-end prediction loss; and sum the weighted local prediction losses and the end-to-end prediction losses to obtain the joint prediction loss.

[0268] Optionally, the training module 1020 is used to backpropagate the joint prediction loss to obtain the model parameter gradients corresponding to the plurality of performance modeling layers and the compensation layer; wherein each of the local prediction losses participates in the calculation of the model parameter gradients of the corresponding performance modeling layer at least, and the end-to-end prediction loss participates in the calculation of the model parameter gradients of the plurality of performance modeling layers and the compensation layer; and the model parameters of the plurality of performance modeling layers and the compensation layer are updated according to the model parameter gradients.

[0269] Optionally, the training module 1020 is configured to use an optimizer to update the model parameters of the initial performance evaluation model according to the model parameter gradient; use the updated model parameters to re-obtain each of the predicted local performance evaluation results and the predicted end-to-end performance evaluation results, and redetermine the joint prediction loss; if the redetermined joint prediction loss does not meet the training stopping condition, repeatedly update the model parameters until the joint prediction loss meets the training stopping condition.

[0270] Optionally, the sample acquisition module 1010 is used to deploy the target model in the sample running environment, measure the running time consumed by the target model from the start of the preset running task to the completion of the preset running task, obtain the end-to-end delay measurement value, and use the end-to-end delay measurement value as the end-to-end performance measurement value.

[0271] The sample acquisition module 1010 is further configured to deploy the logic processing modules corresponding to each performance modeling layer in the sample running environment, measure the running time consumed by each logic processing module to complete the corresponding processing task, obtain the local delay measured value corresponding to each logic processing module, and use each local delay measured value as the corresponding local performance measured value.

[0272] The training device 1000 of the performance evaluation model disclosed herein is used to execute the training method of the performance evaluation model as described in the above embodiments. Its specific processing procedure is the same as that of the training method of the performance evaluation model in the above embodiments, and will not be repeated here.

[0273] This disclosure also provides an electronic device 1100, including:

[0274] At least one processor, and,

[0275] A memory that is communicatively connected to at least one processor; wherein,

[0276] The memory stores instructions that are executed by at least one processor to cause the at least one processor to perform the method as described in any of the above embodiments of this application when executing the instructions.

[0277] The following is combined Figure 11 The hardware structure of the electronic device is described in detail. The electronic device includes: a processor 1110, a memory 1120, an input / output interface 1130, a communication interface 1140, and a bus 1150.

[0278] The processor 1110 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this disclosure.

[0279] The memory 1120 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 1120 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1120, and the processor 1110 calls and executes the model performance evaluation method or the performance evaluation model training method of the embodiments of this disclosure.

[0280] The input / output interface 1130 is used to implement information input and output;

[0281] The communication interface 1140 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.); and the bus 1150 is used to transmit information between the various components of the device (such as processor 1110, memory 1120, input / output interface 1130 and communication interface 1140).

[0282] The processor 1110, memory 1120, input / output interface 1130 and communication interface 1140 are connected to each other within the device via bus 1150.

[0283] This application also provides a computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement the model performance evaluation method or the training method for the performance evaluation model described above, which will not be elaborated here.

[0284] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in this disclosure and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “including,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0285] It should be understood that in this disclosure, "at least one item" means one or more, and "more than one" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0286] It should be understood that in the description of the embodiments of this disclosure, "multiple" means two or more, "greater than", "less than", "exceeding" etc. are understood to exclude the number itself, and "above", "below", "within" etc. are understood to include the number itself.

[0287] In the several embodiments provided in this disclosure, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0288] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0289] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0290] It should also be understood that the various implementation methods provided in this disclosure can be combined arbitrarily to achieve different technical effects.

[0291] The above is a detailed description of the embodiments of this disclosure. However, this disclosure is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this disclosure. All such equivalent modifications or substitutions are included within the scope defined by the claims of this disclosure.

Claims

1. A method for evaluating the performance of a model, characterized in that, The method includes: Obtain the configuration parameters of the target model and the operating environment to be evaluated; The configuration parameters are input into the performance evaluation model, wherein the performance evaluation model includes multiple performance modeling layers and compensation layers connected in sequence. Each performance modeling layer corresponds to multiple logical processing modules executed in sequence during the operation of the target model. The multiple performance modeling layers are connected in sequence according to the execution order of the corresponding logical processing modules. The configuration parameters are input into the multiple performance modeling layers for step-by-step feature processing, wherein the intermediate feature vector output by the previous performance modeling layer is input into the next performance modeling layer until the intermediate feature vector output by the final performance modeling layer is obtained. The intermediate feature vector output by the final performance modeling layer is input into the compensation layer to compensate for the cumulative error generated by the step-by-step feature processing of the multiple performance modeling layers, thereby obtaining the performance evaluation result of the target model in the operating environment to be evaluated.

2. The method according to claim 1, characterized in that, The configuration parameters include common configuration parameters and hierarchical configuration parameters corresponding to each of the performance modeling layers. The step of inputting the configuration parameters into the multiple performance modeling layers for step-by-step feature processing includes: The common configuration parameters are subjected to feature processing to obtain a common feature vector; Feature processing is performed on the hierarchical configuration parameters corresponding to the initial performance modeling layer to obtain the initial layer configuration feature vector; The input feature vector of the initial performance modeling layer is determined based on the common feature vector and the initial layer configuration feature vector. The input feature vector of the initial performance modeling layer is input into the initial performance modeling layer for feature transformation to obtain the intermediate feature vector output by the initial performance modeling layer.

3. The method according to claim 2, characterized in that, The step of inputting the configuration parameters into the multiple performance modeling layers for step-by-step feature processing also includes: For the current performance modeling layer after the initial performance modeling layer, determine the level configuration parameters corresponding to the current performance modeling layer from the configuration parameters; Perform feature processing on the hierarchical configuration parameters corresponding to the current performance modeling layer to obtain the current layer configuration feature vector; The input feature vector of the current performance modeling layer is determined based on the intermediate feature vector output by the previous performance modeling layer and the configuration feature vector of the current layer. The input feature vector of the current performance modeling layer is input into the current performance modeling layer for feature transformation to obtain the intermediate feature vector output by the current performance modeling layer; The above processing is performed sequentially on each performance modeling layer after the initial performance modeling layer until the intermediate feature vector output by the final performance modeling layer is obtained.

4. The method according to any one of claims 1 to 3, characterized in that, The step of inputting the configuration parameters into the multiple performance modeling layers for step-by-step feature processing also includes: The intermediate feature vectors output by each performance modeling layer are mapped to obtain the local performance evaluation results corresponding to each performance modeling layer. The local performance evaluation results of each module represent the operating performance of the corresponding logic processing module in the operating environment to be evaluated.

5. The method according to claim 2, characterized in that, The step of performing feature processing on the common configuration parameters to obtain a common feature vector includes: Determine the numerical common configuration parameters and the categorical common configuration parameters from the common configuration parameters; The numerical common configuration parameters are normalized, and the normalized numerical common configuration parameters are then feature-mapped to obtain a common numerical feature vector. The categorized common configuration parameters are subjected to discrete value encoding, and the categorized common configuration parameters after discrete value encoding are embedded to obtain a common category feature vector; The common numerical feature vector and the common category feature vector are concatenated to obtain the common feature vector.

6. The method according to claim 3, characterized in that, The feature processing of the hierarchical configuration parameters corresponding to the current performance modeling layer includes: Determine the numerical hierarchical configuration parameters and the categorical hierarchical configuration parameters from the hierarchical configuration parameters; The numerical hierarchical configuration parameters are normalized, and the normalized numerical hierarchical configuration parameters are feature-mapped to obtain hierarchical numerical feature vectors. The categorized hierarchical configuration parameters are subjected to discrete value encoding, and the categorized hierarchical configuration parameters after discrete value encoding are embedded to obtain hierarchical category feature vectors. The hierarchical numerical feature vector and the hierarchical category feature vector are concatenated to obtain the current layer configuration feature vector; The intermediate feature vector output by the previous performance modeling layer and the current layer configuration feature vector are concatenated to obtain the input feature vector of the current performance modeling layer.

7. A training method for a performance evaluation model, characterized in that, The initial performance evaluation model to be trained includes multiple sequentially connected performance modeling layers and compensation layers. Each performance modeling layer corresponds to multiple logical processing modules executed sequentially during the execution of the target model. The multiple performance modeling layers are connected sequentially according to the execution order of the corresponding logical processing modules. The method includes: Obtain the sample configuration parameters of the target model running in the sample running environment, the end-to-end performance label of the target model, and the local performance label of each of the logical processing modules; The sample configuration parameters are input into the multiple performance modeling layers for step-by-step feature processing to obtain the predicted local performance evaluation results corresponding to each performance modeling layer, and the intermediate feature vector output by the final performance modeling layer is obtained. The intermediate feature vector output by the final performance modeling layer is input into the compensation layer to compensate for the cumulative error generated by the step-by-step feature processing of the multiple performance modeling layers, thereby obtaining the predicted end-to-end performance evaluation result. For each performance modeling layer, the corresponding local prediction loss is determined based on the prediction local performance evaluation result corresponding to the performance modeling layer and the local performance label of the logic processing module corresponding to the performance modeling layer. Based on the predicted end-to-end performance evaluation results and the end-to-end performance labels, determine the end-to-end prediction loss; The joint prediction loss is determined based on the local prediction loss and the end-to-end prediction loss, and the model parameters of the initial performance evaluation model are updated based on the joint prediction loss until the training stopping condition is met, thus obtaining the performance evaluation model.

8. The method according to claim 7, characterized in that, Obtaining the end-to-end performance label corresponding to the target model and the local performance labels corresponding to each of the logical processing modules includes: The target model is deployed in the sample runtime environment, and the end-to-end performance of the target model when it completes the preset running task is measured to obtain the measured end-to-end performance value. The end-to-end performance label is determined based on the measured end-to-end performance values. The logic processing modules corresponding to each performance modeling layer are deployed and run in the sample runtime environment. The local performance of each logic processing module when it completes the corresponding processing task is measured to obtain the measured local performance values ​​of each logic processing module. Based on the measured values ​​of each local performance, the local performance labels corresponding to each logic processing module are determined.

9. The method according to claim 7, characterized in that, The step of inputting the sample configuration parameters into the multiple performance modeling layers for step-by-step feature processing includes: Determine the common configuration parameters of the samples and the sample hierarchical configuration parameters corresponding to each of the performance modeling layers from the sample configuration parameters; The common configuration parameters of the samples are subjected to feature processing to obtain the common feature vector of the samples; Feature processing is performed on the sample layer configuration parameters corresponding to the initial performance modeling layer to obtain the sample initial layer configuration feature vector; Based on the common feature vector of the samples and the configuration feature vector of the sample initial layer, the sample input feature vector of the initial performance modeling layer is determined; The sample input feature vector is input into the initial performance modeling layer for feature transformation to obtain the intermediate feature vector output by the initial performance modeling layer; For the current performance modeling layer after the initial performance modeling layer, feature processing is performed on the sample layer configuration parameters corresponding to the current performance modeling layer to obtain the current layer sample configuration feature vector; The sample input feature vector of the current performance modeling layer is determined based on the intermediate feature vector output by the previous performance modeling layer and the sample configuration feature vector of the current layer. The sample input feature vector of the current performance modeling layer is input into the current performance modeling layer for feature transformation to obtain the intermediate feature vector output by the current performance modeling layer. The above feature processing is performed sequentially until the intermediate feature vector output by the final performance modeling layer is obtained.

10. The method according to claim 7, characterized in that, The step of determining the joint prediction loss based on each of the local prediction losses and the end-to-end prediction loss includes: The log-mean square error between each predicted local performance evaluation result and the corresponding local performance label is calculated to obtain the local prediction loss for each local prediction. The log-mean square error between the predicted end-to-end performance evaluation result and the end-to-end performance label is calculated to obtain the end-to-end prediction loss. Determine the weights corresponding to each of the local prediction losses and the end-to-end prediction losses; The weighted local prediction losses and the end-to-end prediction losses are summed to obtain the joint prediction loss.

11. The method according to claim 10, characterized in that, The step of updating the model parameters of the initial performance evaluation model based on the joint prediction loss includes: Backpropagation is performed on the joint prediction loss to obtain the model parameter gradients corresponding to the multiple performance modeling layers and the compensation layer; Wherein, each of the local prediction losses participates at least in the calculation of the model parameter gradient of the corresponding performance modeling layer, and the end-to-end prediction loss participates in the calculation of the model parameter gradient of the multiple performance modeling layers and the compensation layer; The model parameters of the plurality of performance modeling layers and the compensation layer are updated according to the gradient of the model parameters.

12. The method according to claim 11, characterized in that, The step of updating the model parameters of the plurality of performance modeling layers and the compensation layer according to the model parameter gradient includes: The optimizer updates the model parameters of the initial performance evaluation model based on the gradient of the model parameters; The local performance evaluation results of each prediction and the end-to-end performance evaluation results of the prediction are obtained again using the updated model parameters, and the joint prediction loss is redetermined. If the redefined joint prediction loss does not meet the training stopping condition, the model parameters are updated repeatedly until the joint prediction loss meets the training stopping condition.

13. The method according to claim 8, characterized in that, The measurement of the end-to-end performance of the target model when it completes the preset running task, to obtain the measured end-to-end performance value, includes: The target model is deployed in the sample running environment, and the running time consumed by the target model from the start of the preset running task to the completion of the preset running task is measured to obtain the end-to-end latency measured value, and the end-to-end latency measured value is used as the end-to-end performance measured value. The measurement of the local performance of each logic processing module when completing the corresponding processing task yields the measured local performance values ​​for each logic processing module, including: The logic processing modules corresponding to each performance modeling layer are deployed and run in the sample running environment. The running time consumed by each logic processing module to complete the corresponding processing task is measured to obtain the local latency measured value corresponding to each logic processing module. Each local latency measured value is used as the corresponding local performance measured value.

14. A model performance evaluation device, characterized in that, The device includes: The acquisition module is used to acquire the configuration parameters of the target model and the operating environment to be evaluated. A processing module is used to input the configuration parameters into a performance evaluation model, wherein the performance evaluation model includes multiple performance modeling layers and a compensation layer connected in sequence, each performance modeling layer corresponds to multiple logical processing modules executed in sequence during the operation of the target model, and the multiple performance modeling layers are connected in sequence according to the execution order of the corresponding logical processing modules. The processing module is further configured to input the configuration parameters into the plurality of performance modeling layers for step-by-step feature processing, wherein the intermediate feature vector output by the previous performance modeling layer is input into the next performance modeling layer until the intermediate feature vector output by the final performance modeling layer is obtained. The compensation module is used to input the intermediate feature vector output by the final performance modeling layer into the compensation layer to compensate for the cumulative error generated by the step-by-step feature processing of the multiple performance modeling layers, so as to obtain the performance evaluation result of the target model in the operating environment to be evaluated.

15. A training device for a performance evaluation model, characterized in that, The initial performance evaluation model to be trained includes multiple performance modeling layers and compensation layers connected in sequence. Each performance modeling layer corresponds to multiple logical processing modules executed in sequence during the operation of the target model. The multiple performance modeling layers are connected in sequence according to the execution order of the corresponding logical processing modules. The device includes: The sample acquisition module is used to acquire the sample configuration parameters of the target model running in the sample running environment, the end-to-end performance label of the target model, and the local performance label of each of the logic processing modules. The training module is used to input the sample configuration parameters into the multiple performance modeling layers for step-by-step feature processing, obtain the prediction local performance evaluation results corresponding to each performance modeling layer, and obtain the intermediate feature vector output by the final performance modeling layer. The training module is also used to input the intermediate feature vector output by the final performance modeling layer into the compensation layer to compensate for the cumulative error generated by the step-by-step feature processing of the multiple performance modeling layers, and to obtain the prediction end-to-end performance evaluation result. The training module is further configured to determine the corresponding local prediction loss for each performance modeling layer based on the prediction local performance evaluation result corresponding to the performance modeling layer and the local performance label of the logic processing module corresponding to the performance modeling layer. The training module is also used to determine the end-to-end prediction loss based on the prediction end-to-end performance evaluation results and the end-to-end performance labels. The training module is further configured to determine a joint prediction loss based on each of the local prediction losses and the end-to-end prediction losses, and update the model parameters of the initial performance evaluation model based on the joint prediction loss until the training stopping condition is met, thereby obtaining the performance evaluation model.

16. An electronic device, characterized in that, The electronic device includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for establishing communication between the processor and the memory. The program, when executed by the processor, implements the model performance evaluation method as described in any one of claims 1 to 6, or implements the training method for the performance evaluation model as described in any one of claims 7 to 13.

17. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs, which can be executed by one or more processors to implement the model performance evaluation method as described in any one of claims 1 to 6, or to implement the training method for the performance evaluation model as described in any one of claims 7 to 13.

Images