Optimization methods, devices, media and program products for end-side algorithm models

Abstract

The embodiments of the present application provide a method, device, medium and program product for optimizing an end-side algorithm model. The method includes: determining a calculation graph of the algorithm model, the calculation graph including multiple operators; evaluating the hardware performance of the terminal device to determine hardware performance information; generating a scheduling space of the algorithm model based on the hardware performance information; optimizing the scheduling space of the algorithm model to determine parameter information of each operator; configuring the algorithm model based on the parameter information to generate an algorithm model deployed on the end-side. The algorithm model is adjusted in combination with the hardware performance of the terminal device to be deployed, and the scheduling space of the algorithm model is optimized, so that the hardware of the terminal device can be fully called to perform algorithm calculations, meet deployment requirements, and improve processing efficiency.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an optimization method for a terminal-side algorithm model, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] The Large Language Model (LLM) based on the Transformer architecture has attracted deep technical attention worldwide and achieved remarkable results. Its powerful understanding and generation capabilities are unlocking and reshaping our understanding of the internal mechanism of artificial intelligence (AI) and profoundly changing the boundaries of AI application scenarios. The LLM model is a deep learning model trained on massive text data. It can not only generate natural language text, but also deeply understand the meaning of the text and handle various natural language tasks such as text summarization, question answering, translation, etc.

[0003] However, large language models have a huge number of parameters, and usually use servers with powerful computing power to meet their massive computing and memory requirements. The corresponding high inference cost seriously hinders the implementation and promotion of this technology. Compared with traditional server-side LLM deployment, migrating LLM models to personal terminal devices has many significant advantages, such as providing personalized services to users of terminal devices, protecting user privacy, supporting offline processing and low cost.

[0004] However, personal terminal devices have deficiencies in hardware computing power and other aspects, and there are still many problems in deploying large language models on the terminal side. Summary of the invention

[0005] The embodiment of the present application provides a method for optimizing a terminal-side algorithm model to solve problems existing in the terminal-side deployment algorithm model.

[0006] Correspondingly, an embodiment of the present application also provides an electronic device, a computer-readable storage medium and a computer program product to ensure the implementation and application of the above system.

[0007] In order to solve the above problems, an embodiment of the present application discloses a method for optimizing a terminal-side algorithm model, wherein the method includes:

[0008] Determining a computational graph of the algorithm model, wherein the computational graph includes a plurality of operators;

[0009] Evaluate the hardware performance of the terminal equipment to be deployed and determine the hardware performance information;

[0010] Generate a scheduling space for the algorithm model based on the hardware performance information;

[0011] Optimizing the scheduling space of the algorithm model and determining parameter information of each operator;

[0012] The algorithm model is configured based on the parameter information to generate an algorithm model deployed on the terminal side.

[0013] Optionally, the algorithm model includes a plurality of basic algorithm models, and after determining the calculation graph of the algorithm model, the method further includes:

[0014] Simplifying the computation graph to determine a simplified computation graph;

[0015] Each operator is obtained from the simplified calculation graph, each operator is analyzed, and operators with dependencies are fused.

[0016] Optionally, simplifying the computation graph to determine a simplified computation graph includes:

[0017] Traversing the computation graph, analyzing the computation elements corresponding to the operators in the computation graph, wherein the computation elements include at least one of the following: a constant, an expression, and a code;

[0018] Performing at least one of the following simplification operations on the computation graph based on the computation elements: folding constants, eliminating common sub-expressions, eliminating invalid codes, and simplifying valid codes;

[0019] Generates a simplified computational graph.

[0020] Optionally, the analyzing of the operators and fusing the operators having dependencies include:

[0021] Analyze each operator in the computation graph and estimate the computation amount and memory access amount of each operator;

[0022] Analyze the dependency relationship between operators in the computation graph and determine operators with dependency relationship;

[0023] Operator fusion processing is performed based on the calculation amount, memory access amount and dependency relationship of the operators.

[0024] Optionally, the performing operator fusion processing based on the computation amount, memory access amount and dependency of the operator includes:

[0025] The target operator is screened based on the amount of computation and the amount of memory access, and the operators that have dependencies with the target operator are merged.

[0026] Optionally, the evaluating the hardware performance of the terminal device to be deployed to determine the hardware performance information includes:

[0027] Querying hardware parameter information of the terminal device based on device information of the terminal device to be deployed, wherein the hardware parameter information includes at least one of the following: cache hierarchy information, computing unit information, memory bandwidth information, and hardware instruction information;

[0028] A hardware evaluation process is performed based on the hardware parameter information to determine hardware performance information.

[0029] Optionally, performing hardware evaluation processing based on the hardware parameter information to determine hardware performance information includes:

[0030] Performing a hardware test on the terminal device to determine a hardware test result;

[0031] The hardware parameter information is modified based on the hardware test result to determine the hardware performance information.

[0032] Optionally, the generating a scheduling space of an operator based on the hardware performance information includes:

[0033] Determine the operator type of the corresponding operator based on the computation amount, memory access amount and the hardware performance information of each operator;

[0034] Based on the operator type and operator optimization strategy of the operator, a scheduling space of the operator is determined.

[0035] Optionally, the determining the operator type of the corresponding operator based on the computation amount, memory access amount and hardware performance information of each operator includes:

[0036] Determine the computing density of operators based on the computing amount and memory access amount of each operator;

[0037] The operator type is determined based on the computation density and hardware performance information of the operator.

[0038] Optionally, the determining the scheduling space of the algorithm model based on the operator type and the operator optimization strategy of the operator includes:

[0039] Matching an operator optimization strategy of the operator based on the operator type of the operator, the operator optimization strategy including at least one of the following: a loop optimization strategy, an instruction optimization strategy, a kernel optimization strategy, and a storage optimization strategy;

[0040] The scheduling space of the algorithm model is determined based on the operator optimization strategy of each operator.

[0041] Optionally, the optimizing the scheduling space of the algorithm model to determine parameter information of each operator includes:

[0042] The scheduling space of the algorithm model is analyzed based on the optimization model to determine parameter information corresponding to each operator.

[0043] Optionally, analyzing the scheduling space of the algorithm model based on the optimization model to determine parameter information corresponding to each operator includes:

[0044] The optimization model is used to predict the scheduling space of the algorithm model to determine the predicted values ​​of multiple parameters of each operator;

[0045] The parameter prediction values ​​are screened to determine the parameter information corresponding to each operator.

[0046] Optionally, the method further includes: constructing a training set based on a search algorithm, and using the training set to train the optimization model.

[0047] Optionally, the search algorithm includes a genetic algorithm, and constructing a training set based on the search algorithm and using the training set to train the optimization model includes:

[0048] Constructing an initialization data set based on the scheduling space, and initializing an optimization model based on the initialization data set;

[0049] The optimization model is used to predict the scheduling space of the algorithm model to determine the predicted values ​​of multiple parameters of each operator;

[0050] Screening multiple parameter prediction values ​​of the operator based on a search algorithm to form a training set;

[0051] The optimization model is trained based on the training set, and the parameters of the optimization model are adjusted based on the loss function.

[0052] Optionally, the algorithm model includes: a large language model.

[0053] The embodiment of the present application also discloses a method for optimizing a terminal-side algorithm model, wherein the method includes:

[0054] Provide configuration page;

[0055] receiving an optimization request, wherein the optimization request is generated based on a trigger of the configuration page;

[0056] Determine a selected basic large language model and device information based on the optimization request;

[0057] Determining a computation graph of the basic large language model, the computation graph comprising a plurality of operators;

[0058] Determine the terminal device based on the device information, evaluate the hardware performance of the terminal device, and determine the hardware performance information;

[0059] Generate a scheduling space for the basic large language model based on the hardware performance information;

[0060] Optimizing the scheduling space of the basic large language model and determining parameter information of each operator;

[0061] The basic large language model is configured based on the parameter information to generate a large language model deployed on the terminal device.

[0062] The embodiment of the present application also discloses an electronic device, which includes: a processor, and a memory communicatively connected to the processor;

[0063] The memory stores computer-executable instructions;

[0064] The processor executes the computer-executable instructions stored in the memory to implement the method described in the embodiment of the present application.

[0065] The embodiment of the present application further discloses a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the method described in the embodiment of the present application.

[0066] The embodiment of the present application further discloses a computer program product, including a computer program / computer executable instructions, wherein the computer program / computer executable instructions, when executed by a processor in an electronic device, implement the method described in the embodiment of the present application.

[0067] Compared with the prior art, the embodiments of the present application have the following advantages:

[0068] In an embodiment of the present application, a calculation graph of the algorithm model is determined, the calculation graph includes multiple operators, so as to simplify the algorithm model, and then the hardware performance of the terminal device is evaluated to determine the hardware performance information, and the scheduling space of the algorithm model is generated based on the hardware performance information. The algorithm model is adjusted in combination with the hardware performance of the terminal device to be deployed, the scheduling space of the algorithm model is optimized, the parameter information of each operator is determined, the algorithm model is configured based on the parameter information, and the algorithm model deployed on the terminal side is generated, so that the algorithm model can be deployed on the terminal device side, and the hardware of the terminal device can be fully called to perform algorithm calculations, which meets the deployment requirements and improves processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 is a structural diagram of an example of a model optimization system in an embodiment of the present application;

[0070] Figure 2 It is a flowchart of the steps of an embodiment of an optimization method of a terminal-side algorithm model of the present application;

[0071] Figure 3It is a flowchart of the steps of an embodiment of a training method for an optimization model of the present application;

[0072] Figure 4 is a flowchart of another embodiment of an optimization method for a terminal-side algorithm model of the present application;

[0073] Figure 5 is a flowchart of steps of an optional embodiment of another method for optimizing a terminal-side algorithm model of the present application;

[0074] Figure 6 is a flowchart of another embodiment of a method for optimizing a terminal-side algorithm model of the present application;

[0075] Figure 7 It is a schematic diagram of the structure of an exemplary device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0076] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0077] The embodiments of the present application are applied to the scenario of deploying a large model on the terminal side, such as deploying a large language model LLM in a terminal device such as a user's mobile phone. Since the storage, computing power and other resources of the terminal device on the user side are limited, the algorithm model needs to be optimized before being deployed to the terminal device.

[0078] The embodiment of the present application provides a model optimization system, which can optimize the algorithm model in combination with the hardware performance of the terminal device, so that the algorithm model can be adapted and deployed on the terminal device side to provide corresponding processing. Figure 1 ,The model optimization system includes : algorithm layer, model access layer, model quantization layer, front-end optimization module, back-end optimization module, model compilation layer and automatic tuning module.

[0079] Among them, the algorithm layer provides an algorithm model library, a model training unit and a model simplification unit. The algorithm model library stores common basic algorithm models such as the large language basic model; the model training unit is used to perform training and fine-tuning on the basic algorithm model, so as to customize and optimize the basic algorithm model through the algorithm layer, and obtain algorithm models such as the large language model that meet its specific needs; the model simplification unit is used to simplify the algorithm model, which supports a variety of general model miniaturization technologies and can be applied according to needs.

[0080] The model access layer is used to receive trained model files. The algorithm model library of the algorithm layer in the embodiment of the present application provides basic algorithm models. Users can also upload their trained algorithm models to the system through the model access layer and store them in the algorithm model library of the algorithm layer. The model access layer can parse model files trained by various frameworks and uniformly convert them into files of algorithm models that can be recognizable for subsequent optimization.

[0081] The model quantization layer is used to quantize the algorithm model and compress the data volume of the algorithm model so that the data volume of the algorithm model is suitable for deployment on the terminal device side.

[0082] The front-end optimization module is used to perform hardware-independent graph optimization, in which the algorithm model is globally described through a computational graph, so that graph-level optimization can be performed on the front end, which can be applied to various back-end hardware. The embodiment of the present application includes a general cache optimization for reasoning design of the algorithm model, and introduces high-precision cache quantization on this basis, which significantly reduces the occupancy of resident memory while ensuring the reasoning accuracy.

[0083] The back-end optimization module is used to perform hardware-related optimizations and generate high-performance code for specific hardware systems.

[0084] In response to the challenge of too large a search space in the optimization process of specific hardware, the embodiment of the present application introduces an automatic tuning module based on machine learning, which greatly accelerates the efficiency and quality of optimization. The algorithm model generated after a series of optimizations is converted into executable code for the target hardware through the model compilation layer, providing users with an efficient reasoning experience.

[0085] Based on the above model optimization system, the present application embodiment provides a method for deploying a model on a terminal side. Figure 2 shown.

[0086] Step 202: determine a computation graph of the algorithm model, where the computation graph includes multiple operators.

[0087] The system can store algorithm models, which can include one or more basic algorithm models. Taking the large language model as an example, various language processing, AI, question-answering and other large language basic models can be stored. At least one algorithm model can be selected from the basic algorithm models as the algorithm model to be optimized, and then optimized.

[0088] Determine the corresponding computational graph for the algorithm model to be optimized, where the computational graph is a directed acyclic graph used to describe the operations in the algorithm model, which can globally describe the algorithm model. The algorithm graph consists of nodes and edges, where nodes are constructed by operators and edges represent the association relationship between operators. Operators refer to functions or classes used to perform various mathematical operations and operations.

[0089] By analyzing the calculation graph of the algorithm model, the content of each operator, such as the type of operator, etc., can be analyzed to determine the operators with dependencies, and then the operators with dependencies are fused. In an optional embodiment, the calculation graph of at least one basic algorithm model is determined, the operators in the calculation graph are analyzed, and the operators with dependencies are fused, including: determining the calculation graph of at least one basic algorithm model, simplifying the calculation graph, and determining a simplified calculation graph; obtaining each operator from the simplified calculation graph, analyzing each operator, and fusing operators with dependencies. Usually, the algorithm model is relatively large in scale, and the corresponding calculation graph is also relatively complex. In order to facilitate subsequent optimization, the calculation graph can be simplified first, and the calculation elements of each operator in the calculation graph are analyzed to determine the calculation elements that can be simplified and perform simplification processing to obtain a simplified calculation graph.

[0090] In an optional embodiment, the simplification of the computation graph and determination of a simplified computation graph include: traversing the computation graph and analyzing the computation elements corresponding to the operators in the computation graph, wherein the computation elements include at least one of the following: constants, expressions, and codes; performing at least one of the following simplification operations on the computation graph based on the computation elements: folding constants, eliminating common sub-expressions, eliminating invalid codes, and simplifying valid codes; and generating a simplified computation graph. Traversing the computation graph and analyzing the computation elements corresponding to each operator in the computation graph, wherein the functions in the algorithm model can be used as operators, and thus the computation elements corresponding to the functions can be analyzed, wherein the computation elements include at least one of the following: constants, expressions, codes, etc. The computation graph is simplified based on the computation elements, and the simplification operations of the simplification process include at least one of the following: folding constants, eliminating common sub-expressions, eliminating invalid codes, and simplifying valid codes. Folding constants means setting the specified constants in the function to the values ​​corresponding to the constants, thereby eliminating the scheduling space occupied by the constants. Eliminating common subexpressions means eliminating repeated common subexpressions to reduce redundancy. Eliminating invalid code means eliminating invalid code in the algorithm model. Simplifying valid code means simplifying the valid code in the algorithm model, thereby simplifying the computational graph.

[0091] After the calculation graph is optimized, the layout of the calculation graph needs to be optimized due to the differences in hardware of different terminal devices. The data layout of the calculation graph is optimized, including memory alignment optimization, tensor data layout optimization, and NCHW data arrangement optimization.

[0092] Among them, memory alignment optimization refers to aligning the address where the data of the computational graph is stored in the memory with an integer multiple of the data reading byte block. Since the data is read in integer multiples, the data can be read at once after memory alignment, making more efficient use of hardware performance. Among them, memory alignment is related to the location of the data in the memory, and memory alignment is in bytes.

[0093] Tensor data layout optimization refers to optimizing the layout of tensor data corresponding to the computational graph, where a tensor is a multidimensional array, that is, optimizing the layout of multidimensional data in the computational graph, such as setting it to row-first arrangement, column-first arrangement, etc. The row-first arrangement refers to arranging the data in the data group in a row-first manner, and the column-first arrangement refers to arranging the data in the data group in a column-first manner.

[0094] For NCHW data arrangement optimization, where N is batch data, H is height, W is width, and C is channels, NCHW data arrangement optimization is an optimization method for continuously arranging data values ​​of the same channel. It can take advantage of the large memory bandwidth and strong parallelism of GPU (Graphics Processing Unit), and its logical control of memory access and calculation is relatively simple.

[0095] Through the above layout optimization, the calculation graph is more in line with the hardware characteristics of the terminal device.

[0096] Then, each operator is obtained from the simplified calculation graph, and each operator is analyzed to determine the association relationship between operators, such as querying operators with dependencies for fusion, etc. In an optional embodiment, the analysis of each operator and fusion of operators with dependencies include: analyzing each operator in the calculation graph to estimate the computation amount and memory access amount of each operator; analyzing the dependency relationship between each operator in the calculation graph to determine operators with dependencies; and performing operator fusion processing based on the computation amount, memory access amount and dependency relationship of the operators.

[0097] Analyze each operator in the calculation graph and estimate the amount of computation and memory access for each operator, where computation refers to the number of times the operator is calculated, and access refers to the number of times the operator accesses the memory, including the number of times the memory is read and written. Estimate the theoretical amount of computation and memory access for each operator in the calculation graph. The theoretical amount of computation depends on the complexity of the algorithm itself, the size of the input data, and the type of algorithm. The computational amount formula is shown in Formula 1 below:

[0098] Formula 1

[0099] Among them, n represents that there are n different types of operations in the operator. Represents the number of times this type of operation occurs. Represents the amount of computation for this type of operation. Different operations have different calculation formulas. Taking General Matrix Multiplication (GEMM) as an example, its calculation amount is shown in the following formula 2:

[0100] Formula 2

[0101] Among them, M, N, and K represent the dimensions of the two GEMM matrices respectively.

[0102] The theoretical access volume formula is shown in Formula 3 below:

[0103] Formula 3

[0104] Among them, MA represents the total access amount, and Respectively represent the number of input and output tensors of the operator, and Respectively represent the number of dimensions (Rank) of the i-th input and j-th output tensors, d represents the specific dimension, Represents the number of bytes corresponding to the tensor type, such as .

[0105] The above method can be used to estimate the amount of computation and memory access of each operator, and the correlation between operators can be analyzed based on the computation graph, and operators with dependencies can be determined through data dependency analysis. Then, operator fusion processing is performed based on the computation amount, memory access amount and dependency of the operator. Target operators are filtered based on the computation amount and memory access amount, for example, the first target operator whose computation amount is greater than the computation threshold is filtered, and the second target operator whose memory access amount is greater than the memory access threshold is filtered, and operators with dependencies on the target operator are fused to reduce additional memory access overhead.

[0106] Step 204: Evaluate the hardware performance of the terminal device to determine hardware performance information.

[0107] A hardware device database on the terminal side is stored in the system in advance. The hardware device database stores hardware data of various types of terminal devices, such as hardware parameter information of different processors, memories, etc. Therefore, when optimizing the algorithm model, the user can provide the device information of the terminal device to be deployed, such as the device name, CPU (Central Processing Unit) model, etc., and then query the hardware device database based on the device information to determine the hardware parameter information of the terminal device, wherein the hardware parameter information includes at least one of the following: cache hierarchy information, computing unit information, memory bandwidth information, and hardware instruction information. The cache hierarchy information refers to the hierarchy of the cache design in the terminal device, which is related to data reading and writing. The computing unit information refers to the information of the unit used for calculation in the terminal device, such as the computing unit in the CPU. The memory bandwidth information refers to the information of the memory and bandwidth in the terminal device, such as the memory size, quantity, bandwidth size, model, etc. The hardware instruction information refers to the information of the hardware instructions of the terminal device, including CPU instructions, etc. Then, hardware evaluation processing is performed based on the hardware parameter information to determine the hardware performance information. Based on the hardware device database, the hardware structure of the terminal device can be perceived and the hardware performance information can be determined.

[0108] In an optional embodiment, the hardware evaluation process based on the hardware parameter information to determine the hardware performance information includes: performing a hardware test on the terminal device to determine the hardware test result; and correcting the hardware parameter information based on the hardware test result to determine the hardware performance information. Since the hardware parameter information of the hardware device database may be deviated in the terminal device, the terminal device may be subjected to a hardware test, and the hardware parameter information may be corrected based on the test result. In an embodiment of the present application, the device evaluation is performed by remote procedure call (RPC) technology. Among them, RPC is a remote procedure call protocol for realizing inter-process communication across a network. It provides a simple and efficient way to enable different components in a distributed system to call remote functions like calling local functions. Therefore, the user's terminal device can be remotely connected based on the RPC technology, and a hardware performance benchmark test can be performed by calling an interface and executing instructions to determine the hardware test result. Then, the hardware parameter information is corrected based on the hardware test result, such as adjusting the computing unit information, memory bandwidth information, etc.

[0109] Step 206: Generate a scheduling space for the algorithm model based on the hardware performance information.

[0110] The scheduling space can be understood as the space of parameter value ranges of various parameters. Since the algorithm model includes multiple operators, each operator corresponds to one or more parameters, and the parameter value of each parameter can be adjusted within a certain range, the range of each parameter value in various algorithm models can be described by the scheduling space. The execution of the algorithm model in the terminal device is closely related to the hardware, so the scheduling space of the algorithm model can be determined based on the hardware performance information of the terminal device.

[0111] In an optional embodiment, the generation of the scheduling space of the operator based on the hardware performance information includes: determining the operator type of the corresponding operator based on the computational amount, memory access amount and the hardware performance information of each operator; and determining the scheduling space of the operator based on the operator type and operator optimization strategy of the operator. Among them, the computational density of the operator is first determined based on the computational amount and memory access amount of each operator; and the operator type is determined based on the computational density of the operator and the hardware parameter information. Computational density is an important indicator for measuring computing efficiency, which reflects the utilization of computing resources and memory bandwidth by the operator during execution. The formula for computing density in an example is shown in Formula 4 below:

[0112] Formula 4

[0113] Among them, OPD (Operator Density) is the calculation density of the operator, is the number of floating-point calculations of the operator, and MA is the memory access amount of the operator.

[0114] Then, based on the computing density and the hardware performance information of the terminal device, the operator type is determined. Operator types include: memory-intensive and compute-intensive. Access-intensive operators refer to operators with memory access bottlenecks, and compute-intensive operators refer to operators with high compute-intensive characteristics. Operator types are measured by computing power thresholds. Hardware performance information includes memory bandwidth information and computing unit information. The computing unit information includes the peak computing power value, and the memory bandwidth information includes the memory bandwidth value. The computing power threshold of the operator is the peak computing power value / memory bandwidth value. If , then the operator has a memory access bottleneck and is a memory access intensive operator. , then the operator has high computational intensity and is a computationally intensive operator.

[0115] In an optional embodiment, the determination of the scheduling space of the algorithm model based on the operator type and operator optimization strategy of the operator includes: matching the operator optimization strategy of the operator based on the operator type of the operator; and determining the scheduling space of the algorithm model based on the operator optimization strategy of each operator. The operator optimization strategy of the operator is determined by matching the operator optimization strategy based on the operator type of the operator, wherein the access-intensive optimization strategy can be matched for memory-intensive operators, and the computation-intensive optimization strategy can be matched for computation-intensive operators, wherein the access-intensive optimization strategy includes: loop optimization strategy, instruction optimization strategy, and kernel optimization strategy.

[0116] Loop optimization strategy refers to the strategy for optimizing loop calculations in the computational graph, including strategies such as loop expansion, blocking, swapping, splitting, and fusion. Loop optimization can speed up the processing of the computational graph.

[0117] Instruction optimization strategy refers to the strategy for optimizing the processing execution in the computational graph, including instruction optimization for vectorized computing, instruction optimization for tensorized computing, etc., such as the instructions used for optimization.

[0118] Kernel optimization strategy refers to the strategy for optimizing the computing core in the computational graph, including optimization of algorithms, memory layout, workgroup adjustment, etc., such as optimization of convolution operators.

[0119] Storage optimization strategies are strategies for optimizing computational graphs based on storage, including optimization strategies for memory access latency and storage allocation, among which memory access latency is the latency of instructions or memory access, i.e., overlapping memory operations with computations to improve the utilization of memory and computing resources. Storage allocation refers to strategies for optimizing memory allocation, including optimization of storage space for local variables, global variables, and heap variables.

[0120] In some examples, when OPD approaches a computing power threshold, it is necessary to consider both access and computing characteristics for optimization.

[0121] After completing the analysis of the operator optimization strategy, the parameter value range of the parameters in the operator can be determined based on the operator optimization strategy, and then the scheduling space can be determined based on the parameter value range of the parameters in each operator. Among them, the preset optimization template can be applied to the operator based on the operator optimization strategy of the operator to generate a customized optimization sketch (Sketch). The optimization template is a pre-set template for operator optimization. The optimization sketch Sketch does not fill in specific parameter values. Accordingly, the parameter value range of the parameters in the operator is relatively wide, and a massive scheduling space is determined in combination with the optimization strategy.

[0122] Step 208: Optimize the scheduling space of the algorithm model and determine parameter information of each operator.

[0123] Based on the above-determined scheduling space of the algorithm model, the algorithm model can be automatically tuned (Auto Tuning) based on the scheduling space. In the embodiment of the present application, the operator optimization strategy and the scheduling space of the algorithm model are combined to customize the calculation scheduling of the operator to form an optimized scheduling space.

[0124] The scheduling space of the algorithm model can be analyzed based on the optimization model to determine the parameter information corresponding to each operator. The optimization model is used to classify and optimize the scheduling space of the algorithm model. The optimization model can be constructed using classification models such as gradient trees, for example, through the XGBoost (eXtreme Gradient Boosting) model. In some other examples, machine learning algorithms such as MLP (Multi-Layer Perceptron) models and reinforcement learning can also be used to predict the scheduling space. The optimization model can be constructed and trained through the above-mentioned machine learning algorithms, so as to predict the scheduling space based on the trained optimization model.

[0125] Wherein, a training set is constructed based on a search algorithm, and the training set is used to train the optimization model. In the embodiment of the present application, a search algorithm is combined to construct a training set for the optimization model, improve the training set, and improve the accuracy of the model. Wherein, the search algorithm includes a genetic algorithm, and the construction of a training set based on a search algorithm, and the use of the training set to train the optimization model includes: constructing an initialization data set based on the scheduling space, initializing the optimization model based on the initialization data set; using the optimization model to predict the scheduling space of the algorithm model, and determining multiple parameter prediction values ​​of each operator; based on the search algorithm, the multiple parameter prediction values ​​of the operator are screened to form a training set; based on the training set optimization model for training, and the parameters of the optimization model are adjusted based on the loss function.

[0126] Randomly select n scheduling parameters in the scheduling space of the algorithm model, and build an XGBoost initialization data set after compilation and RPC real machine testing. The RPC real machine test is a test based on the RPC protocol calling the terminal device. For example, the data set is ,in, is a feature vector consisting of specific parameters used in different optimization techniques, It is the normalized value of the inference time of the real machine test after compilation. Use this data set to initialize the optimization model such as the XGBoost model and save the model locally. Use the optimization model such as the XGBoost model to predict all scheduling parameters in the entire scheduling space. The smaller the predicted value, the less the model believes the calculation time is, that is, the faster the model processing speed is.

[0127] The embodiment of the present application also selects the first p scheduling parameter values ​​in the output result of the optimization model based on a search algorithm such as a genetic algorithm, and randomly selects a small number of q scheduling parameter values ​​from the entire scheduling space as variant individuals, which together constitute a number of The genetic variation scheduling set is then compiled and tested on a real machine using RPC to obtain the measured data and update the training set.

[0128] The optimized model is then trained using the updated training set, where the model training is performed by minimizing the objective function. The objective function is shown in Formula 5 below:

[0129] Formula 5

[0130] Among them, l is the loss function, and the regularization term It is used to penalize the complexity of the model, as shown in Formula 6 below:

[0131] Formula 6

[0132] Where T is the number of leaves in the tree, and are regularization parameters, which control the complexity of the tree and the regularization of the leaf node weights, is the model parameter.

[0133] The optimization model is trained in the above manner to determine whether the optimization model has converged or reached the maximum iteration threshold (Max trials). If it has converged or reached the preset tuning iteration upper limit, the current model is used as the optimization model to predict the parameter information of the scheduling generation operator with better performance. Otherwise, the above model training process is repeated.

[0134] After completing the model training, the optimization model is used to predict the scheduling space of the algorithm model to determine multiple parameter prediction values ​​of each operator; the parameter prediction values ​​are screened to determine the parameter information corresponding to each operator. The parameter value range corresponding to each operator in the algorithm model, that is, the scheduling parameter value range, is input into the optimization model to determine multiple parameter prediction values ​​of the operator, and then the parameter prediction values ​​are screened, such as screening the parameter values ​​with the first M parameter prediction values, where M is a positive integer, and the arrangement order can be from large to small or from small to large, which is specifically set according to the needs, such as selecting one or more parameter values ​​that exceed the threshold, etc., to determine the parameter information corresponding to each operator.

[0135] Step 210: configure the algorithm model based on the parameter information to generate an algorithm model deployed on the terminal side.

[0136] The operators in the algorithm model are configured based on the parameter information, a terminal algorithm model of the corresponding terminal device is generated, and the terminal algorithm model is deployed to the corresponding terminal device.

[0137] In summary, the calculation graph of the algorithm model is determined, the operators in the calculation graph are analyzed, and the operators with dependencies are integrated to simplify the algorithm model. Then, the hardware performance of the terminal device is evaluated, the hardware performance information is determined, and the scheduling space of the algorithm model is generated based on the hardware performance information. The algorithm model is adjusted in combination with the hardware performance of the terminal device to be deployed, the scheduling space of the algorithm model is optimized, the parameter information of each operator is determined, and the algorithm model is configured based on the parameter information. The algorithm model deployed on the terminal side is generated, so that the algorithm model can be deployed on the terminal device side and can fully call the hardware of the terminal device for algorithm calculation, meet the deployment requirements, and improve processing efficiency.

[0138] The automatic tuning technology based on machine learning and genetic algorithms in the embodiment of the present application greatly improves the search efficiency and can generate high-performance tensor programs for operators in minutes, wherein the tensor program is an effective abstraction for representing operators.

[0139] Based on the above embodiments, a training method for an optimization model is provided. Figure 3 As shown:

[0140] Step 302: construct an initialization data set based on the scheduling space of the algorithm model, and initialize the optimization model based on the initialization data set.

[0141] Randomly select n scheduling parameters in the scheduling space of the algorithm model, and build an initialization data set for an optimization model (such as XGBoost model, MLP model, reinforcement learning model) after compilation and RPC real machine testing. The RPC real machine test is a test based on the RPC protocol calling terminal device.

[0142] Step 304: Use the optimization model to predict the scheduling space of the algorithm model and determine multiple parameter prediction values ​​of each operator.

[0143] The optimization model is used to predict all scheduling parameters in the entire scheduling space. The smaller the predicted value, the less calculation time the model believes, that is, the faster the model processing speed.

[0144] Step 306: Screen the multiple parameter prediction values ​​of the operator based on the genetic algorithm to form a genetic variation scheduling set.

[0145] Based on the search algorithm such as genetic algorithm, the first p scheduling parameter values ​​in the output results of the optimization model are selected, and a small number of q scheduling parameter values ​​are randomly selected from the entire scheduling space as variant individuals, which together constitute the number A set of genetic mutation schedules.

[0146] Step 308: compile the genetic variation scheduling data and perform hardware testing. Update the training set based on the test structure.

[0147] The genetic mutation scheduling set is compiled and tested on a real machine using RPC to obtain actual measured data and update the training set.

[0148] Step 310: training the optimization model based on the training set, and adjusting the parameters of the optimization model based on the loss function.

[0149] The optimization model is trained using the updated training set, wherein the training of the model may be performed by minimizing an objective function.

[0150] Step 312, determining whether the model generation condition is met.

[0151] The model generation condition is that the optimization model converges, or the number of iterations reaches the upper limit. If so, that is, the model generation condition is met, then step 314 is executed. If not, that is, the model generation condition is not met, then return to step 304 and repeat steps 304-310.

[0152] Step 314, generating an optimization model.

[0153] The optimization model is trained in the above manner to determine whether the optimization model has converged or reached the maximum iteration threshold (Max trials). If it has converged or reached the preset tuning iteration upper limit, the current model is used as the optimization model to predict the parameter information of the scheduling generation operator with better performance. Otherwise, the above model training process is repeated.

[0154] The embodiment of the present application is based on optimization models such as the XGBoost model, and combines the idea of ​​genetic variation to automatically tune operators. It can quickly screen out the optimal scheduling strategy from the massive scheduling space and generate high-performance operators.

[0155] Based on the above embodiments, the embodiments of the present application also provide a method for optimizing the terminal-side algorithm model, which can optimize the algorithm model based on the front-end and back-end, so that the algorithm model better meets the needs of the terminal device and can be deployed in the terminal device.

[0156] Reference Figure 4 , showing a step flow chart of another embodiment of an optimization method for a terminal-side algorithm model of the present application.

[0157] Step 402: determine a computational graph of the at least one basic algorithm model.

[0158] Step 404: simplify the computation graph to determine a simplified computation graph.

[0159] Among them, the computation graph is traversed to analyze the computation elements corresponding to the operators in the computation graph, and the computation elements include at least one of the following: constants, expressions, and codes; based on the computation elements, at least one of the following simplification operations is performed on the computation graph: folding constants, eliminating common sub-expressions, eliminating invalid codes, and simplifying valid codes.

[0160] Step 406, obtaining each operator from the simplified calculation graph, analyzing each operator, and fusing operators with dependencies.

[0161] Among them, the analyzing of each operator and fusing operators with dependencies include: analyzing each operator in the calculation graph to estimate the computational amount and memory access amount of each operator; analyzing the dependencies between the operators in the calculation graph to determine operators with dependencies; and performing operator fusion processing based on the computational amount, memory access amount and dependencies of the operators.

[0162] Step 408, querying hardware parameter information of the terminal device based on the device information of the terminal device, wherein the hardware parameter information includes at least one of the following: cache hierarchy information, computing unit information, memory bandwidth information, and hardware instruction information.

[0163] Step 410: Perform hardware evaluation processing based on the hardware parameter information to determine hardware performance information.

[0164] Among them, a hardware test is performed on the terminal device to determine a hardware test result; and the hardware parameter information is corrected based on the hardware test result to determine hardware performance information.

[0165] Step 412: Determine the operator type of the corresponding operator based on the computation amount, memory access amount and the hardware performance information of each operator.

[0166] Among them, determining the operator type of the corresponding operator based on the computing amount, memory access amount and hardware performance information of each operator includes: determining the computing density of the operator based on the computing amount and memory access amount of each operator; determining the operator type based on the computing density of the operator and the hardware performance information.

[0167] Step 414: determine the scheduling space of the operator based on the operator type and operator optimization strategy of the operator.

[0168] Among them, determining the scheduling space of the algorithm model based on the operator type and operator optimization strategy of the operator includes: matching the operator optimization strategy of the operator based on the operator type of the operator; and determining the scheduling space of the algorithm model based on the operator optimization strategy of each operator.

[0169] Step 416: Analyze the scheduling space of the algorithm model based on the optimization model to determine parameter information corresponding to each operator.

[0170] Step 418: configure the algorithm model based on the parameter information to generate an algorithm model deployed on the terminal side.

[0171] Among them, the scheduling space of the algorithm model is analyzed based on the optimization model to determine the parameter information corresponding to each operator, including: using the optimization model to predict the scheduling space of the algorithm model to determine multiple parameter prediction values ​​of each operator; screening the parameter prediction values ​​to determine the parameter information corresponding to each operator.

[0172] For unknown mobile hardware devices and other terminal devices, the embodiment of the present application can detect the hardware system of the terminal device, accurately obtain the core performance parameters of the hardware, including cache hierarchy, peak computing power and memory bandwidth, and generate a reasonable scheduling space based on these indicators. It can also combine the genetic variation idea to train the optimization model, and automatically optimize the scheduling space of the algorithm model based on the optimization model, so as to quickly screen out the preferred scheduling strategy from the massive scheduling space and generate high-performance operators.

[0173] Based on the above embodiments, the embodiments of the present application also provide an optimization method for the terminal-side algorithm model, which can optimize the algorithm model based on the front-end and back-end and combine with automatic tuning technology to make the algorithm model more in line with the needs of the terminal device and can be deployed in the terminal device.

[0174] Reference Figure 5 , showing a step flow chart of another embodiment of an optimization method for a terminal-side algorithm model of the present application.

[0175] Step 502: determine a computational graph of the at least one basic algorithm model.

[0176] Step 504: traverse the computation graph and analyze computation elements corresponding to the operators in the computation graph, where the computation elements include at least one of the following: a constant, an expression, and a code.

[0177] Step 506: Perform at least one of the following simplification operations on the computation graph based on the computation elements: folding constants, eliminating common sub-expressions, eliminating invalid codes, and simplifying valid codes.

[0178] Step 508: Generate a simplified computation graph.

[0179] Step 510: Analyze each operator in the computation graph and estimate the computation amount and memory access amount of each operator.

[0180] Step 512: Analyze the dependency relationship between the operators in the computation graph and determine the operators with dependency relationship.

[0181] Step 514: perform operator fusion processing based on the computational effort, memory access effort, and dependency of the operators.

[0182] Step 516: query the hardware parameter information of the terminal device based on the device information of the terminal device.

[0183] The hardware parameter information includes at least one of the following: cache hierarchy information, computing unit information, memory bandwidth information, and hardware instruction information.

[0184] Step 518: Perform a hardware test on the terminal device and determine the hardware test result.

[0185] Step 520: Modify the hardware parameter information based on the hardware test result to determine hardware performance information.

[0186] Step 522: Determine the computation density of the operator based on the computation amount and memory access amount of each operator.

[0187] Step 524: Determine the operator type based on the computation density and hardware performance information of the operator.

[0188] Step 526: Match the operator optimization strategy of the operator based on the operator type of the operator.

[0189] Step 528, determining the scheduling space of the algorithm model based on the operator optimization strategy of each operator.

[0190] Step 530: Analyze the scheduling space of the algorithm model based on the optimization model to determine parameter information corresponding to each operator.

[0191] Step 532: configure the algorithm model based on the parameter information to generate an algorithm model deployed on the terminal side.

[0192] The embodiments of the present application optimize the access, front-end optimization, and back-end optimization of the algorithm model, and build an integrated end-to-end solution, so that users do not need to pay attention to operator optimization and underlying hardware details. They only need to input the original large language model and hardware parameters to obtain a targeted adaptive and optimized executable file, which can be easily deployed on the target hardware and obtain better reasoning performance.

[0193] Taking the deployment of a large language model as an example, the present application embodiment provides a method for optimizing a terminal-side algorithm model, such as Figure 6 shown.

[0194] Step 602, providing a configuration page, wherein the configuration page is used to manage the algorithm model, including model uploading, selection, optimization and other processing.

[0195] The configuration page provides configuration controls, including upload controls, selection controls, optimization controls, etc.

[0196] Among them, the upload control is used to upload the algorithm model. Taking the large language model as an example, the user can upload the large language model based on which he has trained himself. The system also provides some common large language models for users to choose from. After the configuration page is rendered and displayed on the user side, in response to the triggering of the upload control, the basic algorithm model to be uploaded is determined, and in response to the upload confirmation instruction, the corresponding basic algorithm model is uploaded. Subsequently, the basic algorithm model uploaded by the user can be selected by default, or the algorithm model to be optimized can be determined based on the user's selection.

[0197] The selection control is used to select a basic algorithm model, and a basic algorithm model can be selected from algorithm models provided by the system and uploaded by the user. In response to the triggering of the selection control, a list of basic algorithm models is obtained, the basic algorithm models are displayed, and in response to the algorithm selection instruction, the selected basic algorithm model is determined.

[0198] The optimization control is used to optimize the algorithm model. Based on the optimization control, the device information of the terminal device to be deployed can be provided, such as the device model, device identification and other information, so that the system can determine the terminal device based on the device information and query the hardware device database. In response to the triggering of the optimization control, the device information of the terminal device is obtained, and in response to the optimization confirmation instruction, the optimization of the selected basic algorithm model is started.

[0199] Step 604, receiving an optimization request.

[0200] Step 606: Determine the selected basic large language model and device information based on the optimization request.

[0201] Step 608: Determine a computation graph of the basic large language model, where the computation graph includes multiple operators.

[0202] The operators in the computation graph are analyzed, and the computation graph including multiple operators having dependency relationships is fused.

[0203] Step 610: determine the terminal device based on the device information, evaluate the hardware performance of the terminal device, and determine the hardware performance information.

[0204] Step 612: Generate a scheduling space for the basic large language model based on the hardware performance information.

[0205] Step 614: Optimize the scheduling space of the basic large language model and determine parameter information of each operator.

[0206] Step 616: configure the basic large language model based on the parameter information to generate a large language model deployed on the terminal device.

[0207] The embodiment of the present application integrates hardware perception capabilities into the deployment of the terminal-side algorithm model, which can automatically detect the hardware system of the terminal device and obtain core hardware parameters. It can also complete model optimization in the face of unknown mobile hardware devices and achieve high-performance reasoning.

[0208] Configure automatic tuning technology, use machine learning-based optimization algorithms and genetic mutation ideas to greatly improve the speed of searching for optimal scheduling strategies, and support dynamic adjustment of operator parameters, which is more suitable for the optimization of algorithm models such as large language models.

[0209] It should be noted that the embodiments of the present application may involve the use of user data. In actual applications, user-specific personal data can be used in the scheme described herein within the scope permitted by applicable laws and regulations, subject to the requirements of applicable laws and regulations of the country where the user is located (for example, with the user's explicit consent, effective notification to the user, etc.).

[0210] It should be noted that, for the method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present application are not limited by the described order of actions, because according to the embodiments of the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present application.

[0211] On the basis of the above embodiment, this embodiment further provides an optimization device for a terminal-side algorithm model, which is applied to an electronic device such as a server-side device, and the device includes:

[0212] An operator fusion module, used to determine a calculation graph of the algorithm model, wherein the calculation graph includes multiple operators;

[0213] A hardware perception module, used to evaluate the hardware performance of the terminal device and determine hardware performance information;

[0214] A scheduling space determination module, used to generate a scheduling space of the algorithm model based on the hardware performance information;

[0215] An optimization module, used to optimize the scheduling space of the algorithm model and determine parameter information of each operator;

[0216] The model determination module is used to configure the algorithm model based on the parameter information and generate an algorithm model deployed on the terminal side.

[0217] In summary, the calculation graph of the algorithm model is determined, the operators in the calculation graph are analyzed, and the operators with dependencies are integrated to simplify the algorithm model. Then, the hardware performance of the terminal device is evaluated, the hardware performance information is determined, and the scheduling space of the algorithm model is generated based on the hardware performance information. The algorithm model is adjusted in combination with the hardware performance of the terminal device to be deployed, the scheduling space of the algorithm model is optimized, the parameter information of each operator is determined, and the algorithm model is configured based on the parameter information. The algorithm model deployed on the terminal side is generated, so that the algorithm model can be deployed on the terminal device side and can fully call the hardware of the terminal device for algorithm calculation, meet the deployment requirements, and improve processing efficiency.

[0218] Among them, the algorithm model includes multiple basic algorithm models, and the operator fusion module is also used to simplify the calculation graph and determine the simplified calculation graph; obtain each operator from the simplified calculation graph, analyze each operator, and fuse operators with dependencies.

[0219] The operator fusion module includes: a simplification submodule and a fusion submodule, wherein:

[0220] The simplification submodule is used to traverse the computation graph and analyze the computation elements corresponding to the operators in the computation graph, wherein the computation elements include at least one of the following: constants, expressions, and codes; based on the computation elements, perform at least one of the following simplification operations on the computation graph: folding constants, eliminating common subexpressions, eliminating invalid codes, and simplifying valid codes; and generate a simplified computation graph.

[0221] The fusion submodule is used to analyze each operator in the calculation graph, estimate the calculation amount and memory access amount of each operator; analyze the dependency relationship between the operators in the calculation graph, determine the operators with dependency relationships; and perform operator fusion processing based on the calculation amount, memory access amount and dependency relationship of the operators.

[0222] The fusion submodule is used to screen the target operator based on the amount of calculation and the amount of memory access, and fuse the operators that have a dependency relationship with the target operator.

[0223] The hardware perception module is used to query the hardware parameter information of the terminal device based on the device information of the terminal device to be deployed, and the hardware parameter information includes at least one of the following: cache hierarchy information, computing unit information, memory bandwidth information, and hardware instruction information; perform hardware evaluation processing based on the hardware parameter information to determine hardware performance information.

[0224] The hardware perception module is used to perform a hardware test on the terminal device and determine a hardware test result; and to modify the hardware parameter information based on the hardware test result to determine hardware performance information.

[0225] The scheduling space determination module includes: a type determination submodule and a space determination submodule, wherein:

[0226] The type determination submodule is used to determine the operator type of the corresponding operator based on the calculation amount, memory access amount and hardware performance information of each operator;

[0227] The space determination submodule is used to determine the scheduling space of the operator based on the operator type and operator optimization strategy of the operator.

[0228] The type determination submodule is used to determine the calculation density of the operator based on the calculation amount and memory access amount of each operator; and determine the operator type based on the calculation density and hardware performance information of the operator.

[0229] The space determination submodule is used to match the operator optimization strategy of the operator based on the operator type of the operator, and the operator optimization strategy includes at least one of the following: loop optimization strategy, instruction optimization strategy, kernel optimization strategy, and storage optimization strategy; based on the operator optimization strategy of each operator, the scheduling space of the algorithm model is determined.

[0230] The optimization module is used to analyze the scheduling space of the algorithm model based on the optimization model and determine the parameter information corresponding to each operator.

[0231] The optimization module is used to use the optimization model to predict the scheduling space of the algorithm model and determine multiple parameter prediction values ​​of each operator; and to screen the parameter prediction values ​​to determine parameter information corresponding to each operator.

[0232] It also includes: a training module, which is used to construct a training set based on the search algorithm, and use the training set to train the optimization model.

[0233] The training module is used to construct an initialization data set based on the scheduling space of the algorithm model, initialize the optimization model based on the initialization data set; use the optimization model to predict the scheduling space of the algorithm model and determine multiple parameter prediction values ​​of each operator; screen the multiple parameter prediction values ​​of the operator based on the search algorithm to form a training set; train the optimization model based on the training set, and adjust the parameters of the optimization model based on the loss function.

[0234] The embodiment of the present application is based on optimization models such as the XGBoost model, and combines the idea of ​​genetic variation to automatically tune operators. It can quickly screen out the optimal scheduling strategy from the massive scheduling space and generate high-performance operators.

[0235] For unknown mobile hardware devices and other terminal devices, the embodiment of the present application can detect the hardware system of the terminal device, accurately obtain the core performance parameters of the hardware, including cache hierarchy, peak computing power and memory bandwidth, and generate a reasonable scheduling space based on these indicators. It can also combine the genetic variation idea to train the optimization model, and automatically optimize the scheduling space of the algorithm model based on the optimization model, so as to quickly screen out the preferred scheduling strategy from the massive scheduling space and generate high-performance operators.

[0236] Wherein, the algorithm model includes: a large language model.

[0237] On the basis of the above embodiment, this embodiment further provides another device for optimizing the terminal-side algorithm model, which is applied to electronic devices such as server-side devices, and the device includes:

[0238] The page providing module is used to provide the configuration page;

[0239] A receiving module, used for receiving an optimization request, wherein the optimization request is generated based on a trigger of the configuration page;

[0240] A parameter determination module, configured to determine a selected basic large language model and device information based on the optimization request;

[0241] A model optimization module is used to determine a calculation graph of the basic large language model, wherein the calculation graph includes multiple operators; determine a terminal device based on the device information, evaluate the hardware performance of the terminal device, and determine hardware performance information; generate a scheduling space for the basic large language model based on the hardware performance information; optimize the scheduling space of the basic large language model and determine parameter information of each operator; configure the basic large language model based on the parameter information and generate a large language model deployed on the terminal device.

[0242] The embodiments of the present application optimize the access, front-end optimization, and back-end optimization of the algorithm model, and build an integrated end-to-end solution, so that users do not need to pay attention to operator optimization and underlying hardware details. They only need to input the original large language model and hardware parameters to obtain a targeted adaptive and optimized executable file, which can be easily deployed on the target hardware and obtain better reasoning performance.

[0243] The embodiment of the present application integrates hardware perception capabilities into the deployment of the terminal-side algorithm model, which can automatically detect the hardware system of the terminal device and obtain core hardware parameters. It can also complete model optimization in the face of unknown mobile hardware devices and achieve high-performance reasoning.

[0244] Configure automatic tuning technology, use machine learning-based optimization algorithms and genetic mutation ideas to greatly improve the speed of searching for optimal scheduling strategies, and support dynamic adjustment of operator parameters, which is more suitable for the optimization of algorithm models such as large language models.

[0245] The embodiment of the present application further provides a non-volatile readable storage medium, in which one or more modules (programs) are stored. When the one or more modules are applied to a device, the device can execute instructions (instructions) of each method step in the embodiment of the present application.

[0246] The embodiment of the present application further provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the method described in the embodiment of the present application.

[0247] The embodiment of the present application also provides an electronic device, comprising: a processor, and a memory connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the method described in the embodiment of the present application. In the embodiment of the present application, the electronic device includes a server, a terminal device, and other devices.

[0248] The embodiment of the present application further discloses a computer program product, including a computer program / computer executable instructions, wherein the computer program / computer executable instructions, when executed by a processor in an electronic device, implement the method described in the embodiment of the present application.

[0249] The embodiments of the present disclosure may be implemented as a device configured as desired using any appropriate hardware, firmware, software, or any combination thereof, and the device may include electronic devices such as a server (cluster), a terminal, etc. Figure 7 An exemplary apparatus 700 that can be used to implement various embodiments described in this application is schematically illustrated.

[0250] For one embodiment, Figure 7An exemplary apparatus 700 is shown having one or more processors 702, a control module (chip set) 704 coupled to at least one of the processor(s) 702, a memory 706 coupled to the control module 704, a non-volatile memory (NVM) / storage device 708 coupled to the control module 704, one or more input / output devices 710 coupled to the control module 704, and a network interface 712 coupled to the control module 704.

[0251] The processor 702 may include one or more single-core or multi-core processors, and the processor 702 may include any combination of general-purpose processors or special-purpose processors (such as graphics processors, application processors, baseband processors, etc.). In some embodiments, the device 700 can be used as a server, terminal, or other device described in the embodiments of the present application.

[0252] In some embodiments, the apparatus 700 may include one or more computer-readable media (e.g., memory 706 or NVM / storage device 708) having instructions 714 and one or more processors 702 combined with the one or more computer-readable media and configured to execute the instructions 714 to implement a module to perform the actions described in the present disclosure.

[0253] For one embodiment, the control module 704 may include any suitable interface controller to provide any suitable interface to at least one of the processor(s) 702 and / or any suitable device or component in communication with the control module 704 .

[0254] The control module 704 may include a memory controller module to provide an interface to the memory 706. The memory controller module may be a hardware module, a software module, and / or a firmware module.

[0255] The memory 706 may be used, for example, to load and store data and / or instructions 714 for the device 700. For one embodiment, the memory 706 may include any suitable volatile memory, such as a suitable DRAM. In some embodiments, the memory 706 may include double data rate type four synchronous dynamic random access memory (DDR4 SDRAM).

[0256] For one embodiment, control module 704 may include one or more input / output controllers to provide an interface to NVM / storage device 708 and input / output device(s) 710 .

[0257] For example, NVM / storage 708 may be used to store data and / or instructions 714. NVM / storage 708 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable non-volatile storage device(s) (e.g., one or more hard disk drives (HDDs), one or more compact disk (CD) drives, and / or one or more digital versatile disk (DVD) drives).

[0258] NVM / storage device 708 may include storage resources that are part of the device on which apparatus 700 is installed, or it may be accessible to the device without being part of the device. For example, NVM / storage device 708 may be accessed via (one or more) input / output devices 710 over a network.

[0259] (One or more) input / output devices 710 may provide an interface for the apparatus 700 to communicate with any other appropriate device, and the input / output device 710 may include a communication component, an audio component, a sensor component, etc. The network interface 712 may provide an interface for the apparatus 700 to communicate through one or more networks, and the apparatus 700 may wirelessly communicate with one or more components of a wireless network according to any of one or more wireless network standards and / or protocols, for example, accessing a wireless network based on a communication standard, such as Bluetooth, WiFi, 2G, 3G, 4G, 5G, 6G, etc., or a combination thereof for wireless communication.

[0260] For one embodiment, at least one of the processor(s) 702 may be packaged together with the logic of one or more controllers (e.g., a memory controller module) of the control module 704. For one embodiment, at least one of the processor(s) 702 may be packaged together with the logic of one or more controllers of the control module 704 to form a system-in-package (SiP). For one embodiment, at least one of the processor(s) 702 may be integrated on the same die with the logic of one or more controllers of the control module 704. For one embodiment, at least one of the processor(s) 702 may be integrated on the same die with the logic of one or more controllers of the control module 704 to form a system-on-chip (SoC).

[0261] In various embodiments, the apparatus 700 may be, but is not limited to, a terminal device such as a server, a desktop computing device, or a mobile computing device (e.g., a laptop computing device, a handheld computing device, a tablet computer, a netbook, etc.). In various embodiments, the apparatus 700 may have more or fewer components and / or a different architecture. For example, in some embodiments, the apparatus 700 includes one or more cameras, a keyboard, a liquid crystal display (LCD) screen (including a touch screen display), a non-volatile memory port, multiple antennas, a graphics chip, an application-specific integrated circuit (ASIC), and a speaker.

[0262] Among them, the main control chip can be used as a processor or control module in the detection device, sensor data, location information, etc. are stored in a memory or NVM / storage device, the sensor group can be used as an input / output device, and the communication interface may include a network interface.

[0263] The embodiment of the present application also provides an electronic device, including: a processor; and a memory, on which executable code is stored, and when the executable code is executed, the processor executes one or more methods as described in the embodiment of the present application. In the embodiment of the present application, the memory can store various data, such as target files, file and application association data, and user behavior data, so as to provide a data basis for various processing.

[0264] The embodiments of the present application also provide one or more machine-readable media on which executable codes are stored. When the executable codes are executed, the processor executes one or more methods described in the embodiments of the present application.

[0265] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0266] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0267] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0268] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0269] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0270] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0271] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0272] The above is a detailed introduction to the optimization method, device, medium and program product of an end-side algorithm model provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for optimizing a terminal-side algorithm model, wherein: The method comprises: Determine a computational graph of the algorithm model, wherein the computational graph includes a plurality of operators; Evaluate the hardware performance of the terminal equipment to be deployed and determine the hardware performance information; Generate a scheduling space of the algorithm model based on the hardware performance information, where the scheduling space is a parameter value range of one or more parameters corresponding to each operator in the algorithm model; Using an optimization model to predict the scheduling space of the algorithm model, and determining multiple parameter prediction values ​​of each operator; Screening the parameter prediction values ​​to determine parameter information corresponding to each operator; The algorithm model is configured based on the parameter information to generate an algorithm model deployed on the terminal side.

2. The method according to claim 1, wherein: After determining the calculation graph of the algorithm model, the method further includes: Simplifying the computation graph to determine a simplified computation graph; Each operator is obtained from the simplified calculation graph, each operator is analyzed, and operators with dependencies are fused.

3. The method according to claim 2, wherein: The step of simplifying the computation graph and determining a simplified computation graph includes: Traversing the computation graph, analyzing the computation elements corresponding to the operators in the computation graph, wherein the computation elements include at least one of the following: a constant, an expression, and a code; Performing at least one of the following simplification operations on the computation graph based on the computation elements: folding constants, eliminating common sub-expressions, eliminating invalid codes, and simplifying valid codes; Generates a simplified computational graph.

4. The method according to claim 2, wherein: The analysis of each operator and fusion of operators with dependencies include: Analyze each operator in the computation graph and estimate the computation amount and memory access amount of each operator; Analyze the dependency relationship between operators in the computation graph and determine operators with dependency relationship; Operator fusion processing is performed based on the calculation amount, memory access amount and dependency relationship of the operators.

5. The method according to claim 4, wherein: The operator fusion processing based on the calculation amount, memory access amount and dependency relationship of the operator includes: The target operator is screened based on the amount of computation and the amount of memory access, and the operators that have dependencies with the target operator are merged.

6. The method according to claim 1, wherein: The step of evaluating the hardware performance of the terminal device to be deployed and determining the hardware performance information includes: Querying hardware parameter information of the terminal device based on device information of the terminal device to be deployed, wherein the hardware parameter information includes at least one of the following: cache hierarchy information, computing unit information, memory bandwidth information, and hardware instruction information; A hardware evaluation process is performed based on the hardware parameter information to determine hardware performance information.

7. The method according to claim 6, wherein: The performing hardware evaluation processing based on the hardware parameter information to determine the hardware performance information includes: Performing a hardware test on the terminal device to determine a hardware test result; The hardware parameter information is modified based on the hardware test result to determine the hardware performance information.

8. The method according to claim 5, wherein: The generating a scheduling space of an operator based on the hardware performance information includes: Determine the operator type of the corresponding operator based on the computation amount, memory access amount and the hardware performance information of each operator; Based on the operator type and operator optimization strategy of the operator, a scheduling space of the operator is determined.

9. The method according to claim 8, wherein: The determining the operator type of the corresponding operator based on the computation amount, memory access amount and hardware performance information of each operator includes: Determine the computing density of operators based on the computing amount and memory access amount of each operator; The operator type is determined based on the computation density and hardware performance information of the operator.

10. The method according to claim 8, wherein: The determining the scheduling space of the algorithm model based on the operator type and the operator optimization strategy of the operator includes: Matching an operator optimization strategy of the operator based on the operator type of the operator, the operator optimization strategy including at least one of the following: a loop optimization strategy, an instruction optimization strategy, a kernel optimization strategy, and a storage optimization strategy; The scheduling space of the algorithm model is determined based on the operator optimization strategy of each operator.

11. The method according to claim 1, wherein: Also includes: A training set is constructed based on the search algorithm, and the optimization model is trained using the training set.

12. The method according to claim 11, wherein: The search algorithm includes a genetic algorithm, and the step of constructing a training set based on the search algorithm and using the training set to train the optimization model includes: Constructing an initialization data set based on the scheduling space of the algorithm model, and initializing the optimization model based on the initialization data set; The optimization model is used to predict the scheduling space of the algorithm model to determine the predicted values ​​of multiple parameters of each operator; Screening multiple parameter prediction values ​​of the operator based on a genetic algorithm to form a training set; The optimization model is trained based on the training set, and the parameters of the optimization model are adjusted based on the loss function.

13. A method for optimizing a terminal-side algorithm model, wherein: The method comprises: Provide configuration page; receiving an optimization request, wherein the optimization request is generated based on a trigger of the configuration page; Determine a selected basic large language model and device information based on the optimization request; Determining a computation graph of the basic large language model, the computation graph comprising a plurality of operators; Determine the terminal device based on the device information, evaluate the hardware performance of the terminal device, and determine the hardware performance information; Generate a scheduling space of the basic large language model based on the hardware performance information, where the scheduling space is a parameter value range of one or more parameters corresponding to each operator in the algorithm model; Using an optimization model to predict the scheduling space of the algorithm model, and determining multiple parameter prediction values ​​of each operator; Screening the parameter prediction values ​​to determine parameter information corresponding to each operator; The basic large language model is configured based on the parameter information to generate a large language model deployed on the terminal device.

14. An electronic device, wherein: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 13.

15. A computer-readable storage medium, wherein: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 13 when executed by a processor.

16. A computer program product comprising a computer program / computer executable instructions, wherein: When the computer program / computer executable instructions are executed by a processor in an electronic device, the method according to any one of claims 1 to 13 is implemented.

Images