Model running control method and electronic device

Abstract

This application discloses a model operation control method and an electronic device. The method includes: determining a target inference stage to be executed by the model, wherein the target inference stage is any inference stage in the model's inference process; determining the resource requirements of the target inference stage; determining a target processor from at least one processor based on the resource requirements of the target inference stage and the performance characteristics of at least one processor, wherein different processors have different performance characteristics; and using the target processor to execute the inference of the target inference stage.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a model operation control method and electronic device. Background Technology

[0002] With the continuous development of artificial intelligence technology, model-based inference tasks are becoming increasingly common. However, model inference requires relatively high computing power and power consumption. In particular, when deploying and running intelligent models on electronic devices such as laptops, the computing power and power consumption of these devices face severe challenges. Therefore, how to run models reasonably to balance the performance and energy consumption of model inference is a technical problem that needs to be solved by those skilled in the art. Summary of the Invention

[0003] On the one hand, this application provides a model operation control method, including:

[0004] Determine the target inference stage to be executed by the model, wherein the target inference stage is any inference stage in the model's inference process;

[0005] Determine the resource requirements for the target reasoning phase;

[0006] Based on the resource requirements of the target inference stage and the performance characteristics of at least one processor, a target processor is determined from the at least one processor, and different processors have different performance characteristics;

[0007] The target processor is used to perform the reasoning in the target reasoning phase.

[0008] In one possible implementation, the model execution control method also includes:

[0009] In response to the target processor completing the inference of the target inference stage and the target inference stage not being the last inference stage of the model, the target processor is instructed to store the inference state data inferred in the target inference stage into a shared memory area;

[0010] Wherein, the shared memory area is a memory area accessible to the at least one processor;

[0011] The inference state data is the reference data required to execute the next inference stage in the model after the target inference stage.

[0012] In yet another possible implementation, the execution of the inference phase of the target inference stage using the target processor includes:

[0013] If the target inference stage is not the first inference stage of the model, instruct the target processor to obtain the inference state data inferred from the previous inference stage of the target inference stage from the shared memory area;

[0014] Based on the reasoning state data of the previous reasoning stage, the reasoning of the target reasoning stage is executed using the target processor.

[0015] In yet another possible implementation, determining the resource requirements for the target inference phase includes:

[0016] Determine the resource requirement type for the target inference stage, wherein the resource requirement type is either computationally intensive or memory access intensive.

[0017] The step of determining the target processor from the at least one processor based on the resource requirements of the target inference stage and the performance characteristics of the at least one processor includes:

[0018] In response to the resource requirement type of the target inference stage being compute-intensive, a target processor whose computing performance meets the first set requirement is selected from the at least one processor;

[0019] In response to the resource requirement type of the target inference stage being memory access intensive, a target processor whose power consumption performance meets the second set requirements is selected from at least one processor.

[0020] In yet another possible implementation, the model operation control method also includes:

[0021] In response to determining the target inference stage, the current resource status information of the electronic device is obtained, and the resource status information is used to characterize at least the current load status of each processor in the electronic device;

[0022] The step of determining the target processor from the at least one processor based on the resource requirements of the target inference stage and the performance characteristics of the at least one processor includes:

[0023] Based on the resource requirements of the target inference stage, the performance characteristics of at least one processor, and the resource status information, a target processor is determined from the at least one processor.

[0024] In another possible implementation, determining the target processor from the at least one processor based on the resource requirements of the target inference stage and the performance characteristics of the at least one processor includes:

[0025] Based on the resource requirements of the target inference stage and the performance characteristics of at least one processor, a target processor and suitable first operating parameters for the target processor are determined from the at least one processor.

[0026] The execution of the inference phase of the target inference using the target processor includes:

[0027] The target processor is controlled to perform the inference of the target inference phase using the first operating parameters.

[0028] In yet another possible implementation, the model operation control method also includes:

[0029] During the inference process of the target inference stage executed by the target processor, the current power supply capability information of the electronic device is obtained. The power supply capability information is used to characterize at least one of the power supply type, available power, and power supply parameters of the electronic device.

[0030] Based on the power supply capacity information and the resource requirements of the target inference stage, determine the second operating parameters suitable for the target processor;

[0031] Set the operating parameters of the target processor to the second operating parameters.

[0032] In another possible implementation, the resource requirements of the target inference phase include: restricted operating parameters for at least one operator of the target inference phase on different processors, the restricted operating parameters being determined through testing, and the highest operating parameters allowed by the processor to run the operator;

[0033] The step of determining suitable second operating parameters for the target processor based on the power supply capacity information and the resource requirements of the target inference stage includes:

[0034] Determine the target operator currently being executed in the target inference phase;

[0035] Based on the power supply capacity information and the limiting operating parameters of the target operator on the target processor, a second set of suitable operating parameters for the target processor is determined.

[0036] In yet another possible implementation, the model operation control method also includes:

[0037] During the inference process of the target processor executing the target inference stage, at least one of the current temperature information of the target processor and the performance energy efficiency level of the electronic device is obtained;

[0038] The step of determining suitable second operating parameters for the target processor based on the power supply capacity information and the resource requirements of the target inference stage includes:

[0039] Based on at least one of the temperature information and performance efficiency level, as well as the power supply capacity information and the resource requirements of the target inference stage, a second set of suitable operating parameters for the target processor is determined.

[0040] In another aspect, this application also provides an electronic device, including: a controller and at least one processor, wherein different processors have different performance characteristics;

[0041] The controller is configured to: determine the target inference stage to be executed by the model, wherein the target inference stage is any inference stage in the model's inference process; determine the resource requirements of the target inference stage; determine a target processor from the at least one processor based on the resource requirements of the target inference stage and the performance characteristics of at least one candidate processor; and instruct the target processor to execute the inference of the target inference stage.

[0042] The processor is configured to perform inference in the target inference phase in response to an instruction from the controller. Attached Figure Description

[0043] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0044] Figure 1 A flowchart illustrating the model operation control method provided in this application;

[0045] Figure 2 Another flowchart illustrating the model operation control method provided in this application;

[0046] Figure 3 Another flowchart illustrating the model operation control method provided in this application;

[0047] Figure 4 This diagram illustrates the data interaction between processors when different inference stages of a large language model are executed using different processors.

[0048] Figure 5 Another flowchart illustrating the model operation control method provided in this application;

[0049] Figure 6 Another flowchart illustrating the model operation control method provided in this application;

[0050] Figure 7 Another flowchart illustrating the model operation control method provided in this application;

[0051] Figure 8 The diagram illustrates an example of frequency adjustment for the pre-filling and decoding stages of a large language model executed by the processor under different power supply states.

[0052] Figure 9 A schematic diagram of the component architecture of the electronic device provided in this application;

[0053] Figure 10 This is a schematic diagram of another component architecture of the electronic device provided in this application. Detailed Implementation

[0054] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is only for explaining specific embodiments and is not intended to limit the application. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0055] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0056] like Figure 1 This illustration shows a flowchart of a model operation control method provided in this application. This embodiment can be applied to electronic devices, such as laptops, tablets, or desktop computers, or other types of electronic devices, without limitation.

[0057] The method in this embodiment may include:

[0058] S101, Determine the target inference stage to be executed by the model.

[0059] The target reasoning stage can be any reasoning stage in the model's reasoning process.

[0060] Depending on the type of model, the number and types of reasoning stages that can be divided into in the reasoning process will also vary.

[0061] For example, for transformer-based models such as large language models, the model inference process can be divided into a prefill stage and a decoding stage.

[0062] For example, the speech processing process of a speech processing model can be divided into an encoding stage and a decoding stage.

[0063] S102, Determine the resource requirements for the target reasoning phase.

[0064] Among them, the resource requirements in the target reasoning stage can characterize the type of resource requirements, the amount of resource requirements, and the available resource constraints.

[0065] It is understandable that the inference characteristics of different inference stages of a model differ, and therefore, the performance and power consumption requirements of different inference stages will also differ, resulting in different resource requirements for different inference stages. The resource requirements of the target inference stage may be related to the inference characteristics of the target inference stage and the operators that need to be run. Therefore, by combining the inference characteristics of the target inference stage or the running characteristics of pre-tested operators, the resource requirements of the target inference stage can be determined and stored in advance.

[0066] S103, Based on the resource requirements of the target inference stage and the performance characteristics of at least one processor, determine the target processor from at least one processor.

[0067] In this application, the processor can be any processor capable of running models, such as a neural processing unit (NPU), a graphics processing unit (GPU), or a tensor processing unit (TPU); it can also be a central processing unit (CPU), etc., without any specific limitations.

[0068] In this application, different processors exhibit different performance characteristics, which are related to their computing power (related to computational performance) and power consumption. For example, GPUs have higher computing power but also relatively higher power consumption; NPU processors have relatively lower power consumption, but their computing power is typically lower than that of GPUs. Of course, "different processors" can refer to different types of processors or processors of the same type but with different architectures.

[0069] The target processor is the processor among the at least one processor that can meet the resource requirements.

[0070] S104, Utilize the target processor to perform the inference in the target inference phase.

[0071] For example, an electronic device instructs the target processor to execute the target inference phase of the model, so that the target processor performs the inference of the target inference phase.

[0072] In this application, the electronic device can determine the target inference stage of the model and schedule the target inference stage to be executed by different processors through a controller other than the processor. The controller can be other processors or control components besides multiple processors. For example, the processor can include processors such as NPU and GPU, while the controller can be a CPU.

[0073] As can be seen from the above, after determining the target inference stage to be executed by the model, this application can select a target processor based on the resource requirements of the target inference stage and the performance characteristics of at least one processor, so that the performance characteristics of the selected target processor can meet the resource requirements of the target inference stage. This realizes that during the model operation, processors with different performance characteristics are reasonably selected to run different inference stages according to the resource requirements of different inference stages of the model. In this way, the different performance and energy consumption requirements of different inference stages of the model can be taken into account to run different inference stages reasonably, thereby achieving a more reasonable model operation.

[0074] In this application, there are several possibilities for determining the specific implementation of the target processor based on the resource requirements of the target inference stage. The method for determining the target processor will also differ depending on the specific information regarding the resource requirements.

[0075] The following example illustrates one possible implementation method, combined with... Figure 2 Please provide an explanation. For example... Figure 2 This illustrates another flowchart of the model operation control method provided in this application. The method in this embodiment may include:

[0076] S201, Determine the target inference stage to be executed by the model.

[0077] The target reasoning stage can be any reasoning stage in the model's reasoning process.

[0078] S202, Determine the type of resource requirements for the target reasoning phase.

[0079] This resource requirement type falls into either the compute-intensive or memory-access-intensive category.

[0080] Specifically, the resource requirement type for the target inference stage is described as compute-intensive, meaning that the target inference stage has a relatively higher demand for processor computing resources. The resource requirement type for the target inference stage is described as memory-intensive (also known as memory access-intensive), meaning that the target inference stage has a relatively higher demand for processor memory resources.

[0081] For example, if the computational load involved in a particular inference phase of the model is relatively high, while the memory access volume is relatively low, then the inference task in that phase is a computationally intensive task, meaning the resource requirement type for that phase is computationally intensive. Conversely, if the computational load involved in that phase is relatively low, but memory access is frequent, then the inference task in that phase is a memory-intensive task, meaning the resource requirement type for that phase is memory-intensive.

[0082] To facilitate understanding, let's take a large language model as an example:

[0083] The inference phase of a large language model is divided into a pre-filling phase and a decoding phase. The pre-filling phase processes the entire input prompt word, which involves relatively high computational cost and parallelism; therefore, its resource requirements are computationally intensive. The decoding phase, on the other hand, generates subsequent tokens autoregressively based on the intermediate data generated in the pre-filling phase and the first output token. This phase involves relatively less computation but has frequent memory access; therefore, its resource requirements are memory-intensive.

[0084] S203, in response to the resource requirement type of the target inference stage being compute-intensive, select a target processor from at least one processor whose computing performance meets the first set requirements.

[0085] Among them, the processor's computing performance is used to characterize the processor's computing power.

[0086] The first setting requirement can be set as needed. For example, the target processor whose computing performance meets the first setting requirement can be the target processor with the highest computing performance among the at least one processor. Or, the target processor whose computing performance meets the first setting requirement can be the processor that is in the first set number of positions (such as the first two or three positions) of the at least one processor in descending order of computing performance.

[0087] S204, in response to the resource requirement type of the target inference stage being memory access intensive, select a target processor from at least one processor whose power consumption performance meets the second set requirements.

[0088] In this context, processor power consumption performance is used to characterize the amount of computation a processor can perform per unit of energy consumed. Therefore, the more computations a processor can perform per unit of energy consumed, the better its performance, meaning lower power consumption.

[0089] The second setting requirement can also be set as needed. For example, the target processor that meets the second setting requirement can be: the target processor with the best functional performance; or the processor that ranks in the top two sets (such as the top three) according to power consumption performance from high to low.

[0090] To make it easier to understand, let's illustrate with a simple example:

[0091] Taking the large language model as an example again, and assuming that electronic devices have both NPU and GPU processors.

[0092] Since the pre-filling stage is a computationally intensive inference stage, it requires a lot of computing resources and is sensitive to latency. Therefore, a GPU with relatively high computing power can be selected as the target processor to ensure the inference efficiency of pre-filling computation and reduce the time required for the model to infer the first word.

[0093] The decoding stage is a memory-intensive inference stage with relatively low computational load but high memory access frequency. In this case, in order to reduce the power consumption of the electronic device's inference model task, the NPU, which has relatively low power consumption, can be selected as the target processor.

[0094] Of course, this is just a simple example. In practical applications, there may be other types of processors in the same device. Moreover, with the continuous development of computing, there may be processors of the same type but different architectures in the same device. In this case, the target processor can be selected by comprehensively considering the performance characteristics of all processors and the resource requirements of the target inference stage. The specifics will not be elaborated here.

[0095] S205, the target processor is used to perform the inference of the target inference phase.

[0096] In this embodiment, the resource requirements of the model's inference phase are divided into two types: computationally intensive and access-intensive. Since the resource requirements of the inference phase can intuitively reflect the inference phase's demand for computing power and power consumption performance, based on this, and combined with the resource requirements of the target inference phase to be executed by the model, it is possible to more reasonably and efficiently select a target processor whose computing performance or power consumption performance meets the set requirements to process the target inference phase. Thus, it is possible to more efficiently and reasonably determine the processor suitable for different inference phases during the model processing.

[0097] It's understandable that the various inference stages of a model have a sequential order, and each subsequent inference stage may need to rely on various data generated during the execution of the previous inference stage. If two adjacent inference stages do not run on the same processor, then the various data generated by the previous inference stage need to be transferred to the processor running the subsequent inference stage, which may result in relatively high complexity.

[0098] Therefore, in order to reduce the complexity of running different inference stages on different processors, this application can also set up a shared memory area for at least one processor, which is a memory area accessible to each processor.

[0099] The GPU and NPU can establish a connection with the shared memory area through a high-speed bus and access the shared memory area based on Direct Memory Access (DMA) technology to efficiently write or read data from the shared memory area and achieve fast data transfer.

[0100] Based on this, in response to the target processor completing the inference of the target inference stage and the target inference stage not being the last inference stage of the model, the electronic device can also instruct the target processor to store the inference state data generated by the target inference stage into a shared memory area.

[0101] The inference state data is the reference data required to execute the next inference stage in the model, which is located after the target inference stage.

[0102] For example, the inference state data may at least include: the state data inferred by the target inference stage that needs to be provided to the next inference stage. Considering that some models' inference stages also generate partial results from the model's corresponding inference results, the inference state data may also include the output results inferred by the target inference stage. Here, the output results are a portion of the inference results that the model needs to infer.

[0103] For ease of understanding, we will continue to use the reasoning of the large language model as an example.

[0104] The pre-filling stage requires parallel processing of the input sequence (such as prompt words), computation of a key-value cache (KVCache), and generation of the first token to be output. The key-value cache can include a key matrix (K vector) and a value matrix (V vector). Therefore, the inference state data output in the pre-filling stage includes: the output result generated in the pre-filling stage (i.e., the first token); and the key-value cache serving as state data.

[0105] During the decoding phase, based on the first token, reasoning is continued using the key-value cache generated in the pre-filling phase to deduce each subsequent token.

[0106] Furthermore, if the target inference stage is not the first inference stage of the model, then the processor needs to rely on the inference state data derived from the previous inference stage to execute the target inference stage. Therefore, to more conveniently obtain the inference state data derived from the previous inference stage, if the target inference stage is not the first inference stage of the model, the target processor can be instructed to obtain the inference state data derived from the previous inference stage of the target inference stage from the shared memory area. Accordingly, based on the inference state data corresponding to the previous inference stage, the target processor executes the inference of the target inference stage.

[0107] To facilitate understanding, let's take the resource requirement of the target resource as an example, combined with... Figure 3 Please provide an explanation. For example... Figure 3 This illustration shows another flowchart of the model operation control method provided in this application. The method in this embodiment may include:

[0108] S301, Determine the target inference stage to be executed by the model.

[0109] The target reasoning stage can be any reasoning stage in the model's reasoning process.

[0110] S302, Determine the type of resource requirements for the target reasoning phase.

[0111] This resource requirement type falls into either the compute-intensive or memory-access-intensive category.

[0112] S303, Based on the resource requirement type of the target inference stage and the performance characteristics of at least one processor, determine the target processor from the at least one processor.

[0113] Different processors have different performance characteristics.

[0114] The above steps can be found in the relevant descriptions of the previous embodiments, and will not be repeated here.

[0115] S304. Determine whether the target inference stage is the first inference stage of the model. If yes, proceed to step S305; otherwise, proceed to step S306.

[0116] S305, Execute the inference of the target inference stage using the target processor, and proceed to step S308.

[0117] S306, instructing the target processor to obtain the inference state data inferred from the previous inference stage of the target inference stage from the shared memory area.

[0118] The shared memory area is a memory area accessible to the at least one processor.

[0119] The reasoning state data from the previous reasoning stage serves as the reference data required to execute this target reasoning stage.

[0120] S307, based on the inference state data of the previous inference stage, uses the target processor to execute the inference of the target inference stage.

[0121] S308, in response to the target processor completing the inference of the target inference stage and the target inference stage not being the last inference stage of the model, instructing the target processor to store the inference state data of the target inference stage into a shared memory area.

[0122] The reasoning state data of this target reasoning stage serves as the reference data required for the next reasoning stage to execute reasoning.

[0123] After completing step S308, this application may return to step S301 to determine the next inference stage to be executed.

[0124] Of course, if the target inference stage is the last inference stage of the model, then the output of the target inference stage can be output directly without returning to the execution step S301.

[0125] It is understood that this embodiment solves the problem of data isolation between different processors by setting up a shared memory area as a bridge for data interaction between different processors. When a processor needs to execute an inference stage of the model, if the inference stage is not the first inference stage of the model, the electronic device can directly access the shared memory area to obtain the context information inferred from the previous inference stage without having to transfer the context information inferred from the previous inference stage to the processor currently executing the inference stage through the CPU or other controller. This reduces the data transmission overhead of the electronic device and improves data transmission efficiency.

[0126] To facilitate understanding, we will introduce an application scenario below. For ease of description, we will still use a large language model as the model to be run, and take processors including GPUs and NPUs as examples. Figure 4 This diagram illustrates an example of data interaction between processors when different inference stages of a large language model are executed using different processors.

[0127] like Figure 4As can be seen, since large language models include a pre-filling stage and a decoding stage, and as previously mentioned, the pre-filling stage requires significant computational power, its inference can be performed on a GPU. After the GPU completes the inference in the pre-filling stage, it can write the key-value cache (KV cache) and the first token inferred from the pre-filling stage to the shared memory area. Of course, the GPU can also synchronously output this first token.

[0128] Based on this, upon determining the decoding phase to be executed, the electronic device can instruct the NPU to perform inference for the decoding phase of the large language model and to retrieve context data from the shared memory area for the pre-filling phase output. Based on this, the NPU can obtain the first token and the key-value cache from the shared memory, and based on the key-value cache, perform inference for the decoding phase to deduce each subsequent token one by one.

[0129] It is evident that executing the two inference stages of the large language model on GPU and NPU respectively not only allows for the rational use of different processors to perform inference tasks based on the resource requirements of the two inference stages, thus balancing the efficiency and energy consumption of model inference; but also eliminates the need for electronic devices to transfer data inferred from different inference stages between different processors, resulting in lower overhead.

[0130] In the above embodiments of this application, since the processor load varies at different times and the available resource status on the processor varies, the processor's computing power and energy efficiency will also be affected. Based on this, in any of the above embodiments, in response to determining the target inference stage, this application can also obtain the current resource status information of the electronic device. This resource status information is at least used to characterize the current load status of each processor in the electronic device. For example, the processor's resource status information may include status indicator data that can characterize the processor load, such as processor utilization and memory access bandwidth. Of course, the processor's resource status data may also include processor temperature and the electronic device's battery level, which are relevant indicator data that can affect the processor's computing power or energy efficiency, and there are no limitations on this.

[0131] Accordingly, this application can determine the target processor from the at least one processor based on the resource requirements of the target inference stage, the performance characteristics of at least one processor, and resource status information. For example, the target processor can be determined from the at least one processor based on the resource requirement type of the target inference stage, the performance characteristics of at least one processor, and resource status information.

[0132] Among them, determining the target processor by combining resource status information can take into account the current load status and other resource status information of the target processor when selecting the target processor. This can more reasonably select the target processor that is suitable for the resource requirements (resource requirement type) of the target inference stage, reduce the situation of selecting a processor that meets the resource requirements of the target inference stage but has an excessively high load, and reduce the impact on the processor.

[0133] For example, candidate processors can be identified first based on the resource requirements of the target inference stage and the performance metrics of each processor. If the resource status information of a candidate processor indicates that the processor is not suitable for running the target inference stage (e.g., the processor's load is too high), then a new candidate processor can be selected from other processors based on the resource type of the target inference stage. If the resource status information of the candidate processor is suitable for running the target inference stage, then the candidate processor is determined as the target processor.

[0134] It is understandable that if the operating parameters of the processor during the target inference stage are unreasonable, it will not only lead to an imbalance between processor performance and power consumption, but may also cause significant fluctuations in the battery power or power supply capacity of electronic devices, potentially resulting in abnormal situations such as data loss due to insufficient power supply. Therefore, in the above embodiments of this application, this application can determine the operating parameters of the target processor in addition to determining the target processor itself. The following is in conjunction with… Figure 5 Please provide an explanation, such as Figure 5 This illustration shows another flowchart of the model operation control method provided in this application. The method in this embodiment may include:

[0135] S501, Determine the target inference stage to be executed by the model.

[0136] The target reasoning stage can be any reasoning stage in the model's reasoning process.

[0137] S502, determine the resource requirements for the reasoning phase of this objective.

[0138] In this embodiment, resource requirements are used to characterize the resource needs of the target inference stage, reflecting the requirements of the target inference stage for processor operating parameters. Processor operating parameters refer to parameters that affect the performance and energy efficiency of the processor; these parameters are adjustable and can be applied to the processor. For example, processor operating parameters may include, but are not limited to, at least one of voltage and frequency (also known as operating frequency).

[0139] In this embodiment, the resource requirement may include, but is not limited to, one or more of the following: the type of resource requirement in the target inference stage, the degree of resource requirement of the target inference stage for different types of resources, and the limiting operating parameters of at least one operator in the target inference stage on different processors.

[0140] An operator is a basic computational unit used to perform specific mathematical operations or tasks. For example, an operator can be a function or a class. Operators are fundamental units in the computational graph that constitutes models such as neural networks. A model's computational graph typically includes multiple operators, and the types and number of operators required to run vary at different inference stages of the model. For instance, the operations performed by an operator can be matrix multiplication or addition, without restriction.

[0141] The operating parameters of an operator on a processor are the highest operating parameters allowed for that processor to run the operator, determined through testing. For example, taking the processor's operating parameters, including its frequency, as an example, the operating parameters corresponding to an operator for a processor may include a limited operating frequency, which is the highest frequency allowed for the processor to run the operator, determined through testing.

[0142] The highest operating parameters of the operator on each processor can be pre-tested and stored. There are no restrictions on the specific implementation for obtaining the operating parameters of the operator on different processors.

[0143] For example, for a processor, the optimal operating parameters for the operator's operation efficiency can be determined by running the operator on the processor during the startup or idle phase of the electronic device and by continuously adjusting the operating parameters such as the voltage and frequency applied to the processor. These optimal operating parameters are then defined as the limiting operating parameters.

[0144] Alternatively, one could combine different operating parameters used in the past when loading and running the operator on the processor to determine the operating parameters that make the operator run most efficiently, and then define these operating parameters as the limiting operating parameters.

[0145] S503, based on the resource requirements of the target inference stage and the performance characteristics of at least one processor, determine the target processor and suitable first operating parameters for the target processor from the at least one processor.

[0146] The first operating parameter refers to the operating parameters suitable for the target processor to initiate the execution of the target inference phase. As mentioned earlier, the first operating parameter may include a suitable frequency for the target processor, and may also include a suitable voltage for the target processor.

[0147] The specific implementation of determining the target processor can be found in the relevant descriptions in the previous embodiments, and will not be repeated here.

[0148] In this embodiment, the first operating parameters suitable for the target processor can be determined after the target processor has been determined, based on the resource requirements of the target inference stage and the performance characteristics of the target processor; or it can be determined based on the resource requirements of the target inference stage, based on the first operating parameters suitable for the target processor to execute the target inference stage.

[0149] For example, in one instance, the first set of operating parameters suitable for the target processor can be determined based on the resource requirement type of the target inference stage. For instance, the operating parameters corresponding to different resource requirement types of the processor can be pre-configured, and the operating parameters corresponding to the resource requirement type of the target inference stage can be determined as the first set of operating parameters suitable for the processor.

[0150] In another example, suitable first operating parameters for the target processor can be determined based on the limiting operating parameters of the operators that need to be run during the target inference phase. For example, the suitable first operating parameters for the target processor can be determined by combining the limiting operating parameters of the first operator that needs to be run during the target inference phase, or by combining the limiting operating parameters of each operator that needs to be run during the target inference phase. The value of the first operating parameter does not exceed the limiting operating parameter of the first operator or the largest limiting operating parameter among all operators.

[0151] Of course, in practical applications, the two examples above can be combined to determine the first operating parameters suitable for the target processor.

[0152] S504, The target processor is controlled to perform the inference of the target inference phase using the first operating parameters.

[0153] The first running parameter is applied to the target processor so that the target processor runs based on the first running parameter.

[0154] For example, the power management integrated circuit (PMIC) sets the operating parameters of the target processor to the first operating parameters, and then instructs the target processor to perform the inference of the target inference stage, so that the target processor performs the inference of the target inference stage while running with the first operating parameters.

[0155] For example, the first operating parameters may include a first frequency and a first voltage. Since the processor's voltage and operating frequency and other operating parameters can be set by the power management unit (PMU) in the PMIC module, the PMU can be used to set the processor's operating parameters to the first frequency and the first voltage.

[0156] In this embodiment, after determining the target inference stage to be executed by the model, based on the resource requirements of the target inference stage and the performance characteristics of at least one processor, not only can the target processor for processing the target inference stage be determined, but also the first operating parameters suitable for the target processor can be determined. This allows for a more reasonable determination of the operating parameters suitable for the target processor to execute the target inference stage, while taking into account the resource requirements of the target inference stage and reasonably controlling the operation of the processor, thus balancing the performance and energy efficiency of the processor.

[0157] Furthermore, considering that the available power and power supply methods of electronic devices can affect processor performance, potentially impacting the processor's computational power or energy efficiency during the target inference phase, this application can also obtain the current power supply information of the electronic device to more reasonably determine the suitable first operating parameters for the target processor during the target inference phase. For example, in response to determining the target inference phase, the current power supply information of the electronic device can be obtained. Accordingly, this application can determine the suitable first operating parameters for the target processor based on the resource requirements of the target inference phase and the current power supply information of the electronic device.

[0158] The power supply capability information is used to characterize at least one of the following: power supply type, available power, and power supply parameters of the electronic device. The power supply type refers to the type of power source used by the electronic device, such as battery power (DC power) or power supply via a power adapter (AC power). The power supply parameters can include the input power and rated power of the power supply (e.g., the rated power of the power adapter), and may also include the maximum rated power supported by the processor.

[0159] There are several possible implementations for determining the first operating parameters suitable for the target processor based on the current power supply capacity information of the electronic device and the resource requirements of the target inference stage, and no restrictions are imposed on this.

[0160] For example, based on the current power supply capacity of the electronic device and the resource requirements of the target inference stage, a trained predictive model can be used to predict the first suitable operating parameters for the target processor. This predictive model can be a trained large language model or other machine learning model, etc., without limitation.

[0161] For example, this application can pre-set the maximum operating parameters allowed for the processor under different power supply state levels. Accordingly, the current power supply state level of the electronic device can be determined based on power supply capability information, and the first suitable operating parameters for the target processor can be comprehensively determined based on the maximum operating parameters corresponding to the power supply state level and the resource requirement type of the target inference stage.

[0162] The power supply status level is used to characterize the current power sufficiency of electronic devices. For example, the power supply status level can be divided into three levels: sufficient power, limited power, and insufficient power. Of course, there are other ways to classify the power supply status level, and there are no restrictions on this.

[0163] The higher the power supply status level, the more abundant the electrical energy for the electrons, and correspondingly, the higher the maximum operating parameter value (such as the highest frequency) of the target processor. Based on this, when the resource requirements in the target inference stage are the same, the higher the power supply status level, the higher the maximum operating parameter, and therefore, the larger the value of the first suitable operating parameter for the target processor.

[0164] For example, consider the processor's operating parameters, including its frequency.

[0165] Assuming the resource requirement type of the target inference stage is compute-intensive, the power supply status level can be determined based on the power supply capability information of the electronic device, and then the frequency of the target processor can be set to the highest frequency corresponding to that power supply status level.

[0166] Assuming the resource requirement type for the target inference phase is memory-intensive, if the power supply status of the electronic device is determined to be "sufficient" based on the current power supply information, the target processor's frequency can be set to the highest frequency corresponding to sufficient power. If the power supply status is "limited," the target processor's frequency can be reduced based on the highest frequency corresponding to limited power and the target processor's current frequency, setting it below that highest frequency. If the power supply is insufficient, the target processor's frequency can be set to the lowest frequency below that highest frequency, ensuring the target processor can execute the target inference phase.

[0167] Understandably, during the process of controlling the target processor to execute the target inference phase using the first operating parameters, the power supply information of the electronic device may change dynamically, which may cause the operating parameters of the target processor to become unsuitable for executing the inference processing of the target inference phase. For example, if the power supply of the electronic device drops significantly, but the target processor still executes the inference of the target inference phase at a high frequency, the power supply of the electronic device may become unstable or even fail, resulting in data loss or execution abnormalities in the target inference phase.

[0168] Therefore, in order to ensure that the target processor can reliably and stably execute the model's inference phase while taking into account both model performance and energy efficiency, in any of the above embodiments of this application, during the inference process of the target processor executing the target inference phase, the current power supply capability information of the electronic device is obtained. The power supply capability information is used to characterize at least one of the electronic device's power type, available power, and power supply parameters. Based on this, a suitable second operating parameter for the target processor can be determined based on the currently obtained power supply capability information and the resource requirements of the target inference phase; the operating parameters of the target processor can be set to the second operating parameter.

[0169] It is understood that the power supply capability information of the electronic device before the target processor is determined is obtained at a different time than the power supply capability information during the reasoning process of the electronic device executing the target reasoning result. Therefore, the power supply capability information obtained at these two times can be different. For ease of distinction, this application may refer to the power supply capability information of the electronic device obtained in response to the determination of the target reasoning stage as the first power supply capability information; and the power supply capability information obtained during the reasoning process of the target processor executing the target reasoning stage as the second power supply capability information.

[0170] The process of determining the second operating parameter based on the resource requirements of the target inference stage and the power supply capacity information of the electronic device during the execution of the target inference stage by the target processor can be similar to the process of determining the first operating parameter.

[0171] For example, based on the resource requirements and power supply information of the target inference stage, a predictive model can be used to predict the second operating parameters suitable for the target processor.

[0172] For example, the power supply status level of the electronic device can be determined based on the power supply capacity information. Based on the maximum operating parameters corresponding to the power supply status level and the resource requirement type of the target inference stage, the first operating parameters suitable for the target processor can be determined comprehensively.

[0173] In one possible implementation, considering that the resource conditions of the target inference stage differ depending on the operators running in that stage, the allowed operating parameters, such as the frequency at which the target processor runs the operators, also differ. Therefore, in this embodiment, the resource requirements of the target inference stage may include: limited operating parameters for at least one operator of the target inference stage on different processors. These limited operating parameters are the highest operating parameters allowed by the processor to run the operator, determined through testing. For details on determining the limited operating parameters of the operator on different processors, please refer to the relevant descriptions in the preceding embodiments; they will not be repeated here.

[0174] The following is combined Figure 6 This section introduces one possible implementation method. For example... Figure 6 This illustrates another flowchart of the model operation control method provided in this application. The method in this embodiment may include:

[0175] S601, Determine the target inference stage to be executed by the model.

[0176] The target reasoning stage refers to any reasoning stage in the model's reasoning process.

[0177] S602, Determine the resource requirements for the reasoning phase of this objective.

[0178] The resource requirements include, at a minimum, the limiting operating parameters of at least one operator in the target inference phase on different processors.

[0179] Furthermore, the resource requirement may also include the resource requirement type for the target inference stage, such as at least one of computationally intensive and memory-access intensive requirements. Of course, the resource requirement may also include or characterize other information from the preceding embodiments, as detailed in the preceding descriptions, and will not be repeated here.

[0180] S603, based on the resource requirements of the target inference stage and the performance characteristics of at least one processor, determine the target processor and the first operating parameters of the target processor from the at least one processor.

[0181] Different processors have different performance characteristics.

[0182] In this embodiment, the specific implementation of determining the first operating parameter of the target processor can be found in the relevant description of the previous embodiments, and will not be repeated here.

[0183] S604, the target processor is controlled to perform the inference of the target inference phase using the first operating parameters.

[0184] The above steps can be found in the relevant descriptions of the previous embodiments, and will not be repeated here.

[0185] S605: During the execution of the target inference phase by the target processor, information on the current power supply capability of the electronic device is obtained.

[0186] Among them, the power supply capability information is used to characterize the power type, available power, and at least one of the power supply parameters of the electronic device, as detailed in the previous introduction.

[0187] S606, determine the target operator currently being executed in the target inference phase.

[0188] The target operator is the operator currently executed in the target inference phase. Therefore, the target operator can be any one of the at least one operator that needs to be executed in the target inference phase.

[0189] S607, based on the power supply capability information and the limiting operating parameters of the target operator on the target processor, determine the second operating parameters suitable for the target processor.

[0190] Since the limiting operating parameters corresponding to the target operator are pre-tested, the target processor can operate the operator with the highest allowed operating parameters. Therefore, in order to prevent the target processor or other electronic components from being damaged due to excessively high second operating parameters (frequency or voltage), the second operating parameter shall not exceed the limiting operating parameters corresponding to the target operator.

[0191] There are several possibilities for determining this second running parameter. The following are some examples of possible scenarios:

[0192] For example, based on power supply capacity information and the limiting operating parameters corresponding to the target operator, a prediction model can be used to predict the second suitable operating parameters for the target processor.

[0193] For example, as mentioned earlier, the power supply capability information of an electronic device can characterize the power supply state level of the electronic device. Different maximum operating parameters are suitable for the processor under different power supply state levels. Based on this, the power supply state level corresponding to the electronic device is determined based on this power supply capability information. Then, the second operating parameter is determined based on the maximum operating parameter corresponding to the power supply state level and the limiting operating parameter corresponding to the target operator. For instance, if the maximum operating parameter is higher than the limiting operating parameter corresponding to the target operator, the limiting operating parameter of the target operator is determined as the second operating parameter; if the maximum operating parameter is lower than the limiting operating parameter, the maximum operating parameter is determined as the second operating parameter.

[0194] For example, different parameter adjustment strategies can be set for different power supply status levels, with each power supply status level corresponding to a different parameter adjustment strategy. For instance, different parameter adjustment strategies can specify different reduction rates for operating parameters. Accordingly, after determining the power supply status level of the electronic device based on this power supply capability information, the limiting operating parameters of the target operator can be adjusted based on the parameter adjustment strategy corresponding to that power supply status level, and the adjusted limiting operating parameters are determined as the second operating parameters.

[0195] For example, if the power supply status is "sufficient power," then the parameter reduction can be 0, and the limiting operating parameter of the target operator can be determined as the second operating parameter. If the power supply status is "insufficient power," then the parameter reduction can be 50%, and the parameter value obtained after reducing the limiting operating parameter by 50% can be determined as the parameter value of the second operating parameter.

[0196] Of course, there are other ways to determine the second running parameter, and there are no restrictions on this.

[0197] S608, set the target processor's operating parameters to the second operating parameters so that the target processor can continue to execute the inference of the target inference stage using the second operating parameters.

[0198] It should be noted that during the target inference phase of the target processor, step S605 can be executed periodically or irregularly, and subsequent steps can be executed if there are changes in the power supply status information, so as to adjust the operating parameters of the target processor in a timely manner.

[0199] In this embodiment, during the execution of the target inference phase by the target processor, the current power supply status information of the electronic device is also obtained. Based on the power supply status information of the electronic device and the restricted operating parameters allowed on the target processor by the target operator currently being executed in the target inference phase, the operating parameters of the target processor are adjusted. This allows for reasonable adjustment of the target processor's operating parameters during the execution of the target inference phase, which not only ensures stable operation of the target inference phase while taking into account the performance and energy efficiency of the target processor, but also reduces the possibility of abnormal power supply or even data loss of the electronic device due to unreasonable processor operating parameters.

[0200] It is understandable that during the inference phase of the target inference process, the operating parameters suitable for the target processor to perform the target inference phase will vary depending on the temperature of the target processor. For example, if the temperature of the target processor is too high, the frequency of the target processor needs to be reduced to avoid abnormal operation due to overheating.

[0201] Based on this, in any of the above embodiments of this application, the current temperature information of the target processor can be obtained during the inference process of the target inference stage. Accordingly, based on the temperature information, the current power supply capacity information (i.e., the second power supply capacity information), and the resource requirements of the target inference stage, suitable second operating parameters for the target processor can be determined.

[0202] For example, if the temperature information indicates that the target processor's temperature does not exceed a set threshold, a second operating parameter can be determined by combining the power supply capacity information and the resource requirements of the target inference stage. The specific determination of the second operating parameter can be found in the preceding related descriptions. If the temperature information indicates that the target processor's temperature exceeds a set threshold, the second operating parameter of the target processor can be determined as the set minimum operating parameter; alternatively, after determining a candidate operating parameter based on the power supply capacity information and the resource requirements of the target inference stage, a preset proportion of the candidate operating parameter (preset proportion less than 1) can be determined as the second operating parameter.

[0203] Similarly, when an electronic device operates at different performance efficiency levels, the maximum values ​​of the operating parameters that the processor in the electronic device can use will also differ. Therefore, in any of the above embodiments of this application, during the inference process of the target processor executing the target inference stage, this application can also obtain the performance efficiency level of the electronic device. Accordingly, this application can also determine suitable second operating parameters for the target processor based on the performance efficiency level, the current power supply capacity information (i.e., the second power supply capacity information), and the resource requirements of the target inference stage.

[0204] Among them, the performance and energy efficiency level of an electronic device represents the different levels of demand for the performance and energy efficiency of the processor in the electronic device.

[0205] For example, a performance efficiency rating may include at least: a first performance efficiency rating characterizing the processor's highest (or best) performance, a second performance efficiency rating characterizing the processor's highest (or best) energy efficiency, and at least one third performance efficiency rating. Different third performance efficiency ratings represent different degrees of balance between performance and energy efficiency. For instance, the first performance efficiency rating could characterize an electronic device in a high-performance mode; the second performance efficiency rating could characterize an electronic device in a low-power mode; and the third performance efficiency rating could characterize an electronic device in a balanced mode.

[0206] The performance and energy efficiency level of the electronic device can be set by the user, pre-configured, or determined and configured by the electronic device based on its own operating status, without any specific restrictions.

[0207] Different processors have different reference operating parameters at different performance and energy efficiency levels, which are the highest permissible operating parameters for the processor. Based on this, this application can determine the target reference operating parameters corresponding to the target processor based on the performance and energy efficiency level of the electronic device. Accordingly, by combining the current power supply capacity information of the electronic device and the resource requirements of the target inference stage, the operating parameters of the target processor are adjusted based on the target reference operating parameters to obtain the second operating parameters of the target processor, the parameter value of which is lower than the parameter value of the target reference operating parameters.

[0208] Of course, in practical applications, this application can also combine the target processor's current temperature information, at least one of the performance and energy efficiency levels of the electronic device, the current power supply capacity information of the electronic device, and the resource requirements of the target inference stage to determine the second operating parameters suitable for the target processor.

[0209] To make it easier to understand, let's take the processor's operating parameters, including its frequency and voltage, as an example.

[0210] Considering that the processor voltage is related to the processor frequency, this application can determine the frequency of the target processor by combining information such as the resource requirements of the target inference stage, and then find out the appropriate voltage for the processor at that frequency.

[0211] The table below, taking the NPU as an example, shows the appropriate voltage configurations for the NPU at different frequencies.

[0212]

[0213] As can be seen from the above, when the frequency of the target processor is determined to be 1GHz, the voltage required to be applied to the target processor is 0.85V; when the frequency of the target processor is 920MHz, the voltage required to be applied to the target processor is 0.78V.

[0214] Based on this, the following section will take a specific implementation as an example, combined with... Figure 7 Please provide an explanation. For example... Figure 7 This illustrates another flowchart of the model operation control method provided in this application. The method in this embodiment may include:

[0215] S701, Determine the target inference stage to be executed by the model.

[0216] Among them, the target reasoning stage is any reasoning stage in the reasoning process of the model;

[0217] S702, determine the resource requirements for the inference phase of the target and the current first power supply capability information of the electronic device.

[0218] The resource requirements include the resource requirement type for the target inference stage and the limiting operating parameters of at least one operator in the target inference stage on different processors. In this embodiment, the limiting operating parameters may include at least the limiting frequency of the operator on the processor, and of course, may also include the limiting voltage of the operator on the processor.

[0219] The first power supply capability information is used to characterize at least one of the following: power supply type, available power, and power supply parameters of the electronic device.

[0220] S703, based on the resource requirement type of the target inference stage and the performance characteristics of at least one processor, determine the target processor from the at least one processor.

[0221] Different processors have different performance characteristics.

[0222] The above steps can be found in the relevant descriptions of the previous embodiments, and will not be repeated here.

[0223] S704, based on the resource requirement type of the target inference stage and the first power supply capability information, determine the first operating parameters suitable for the target processor.

[0224] The first operating parameters may include a first target frequency and a first voltage suitable for the target processor.

[0225] For example, based on the resource requirement type of the target inference stage and the first power supply capability information, a first frequency suitable for the target processor is determined, and then a first voltage suitable for the target processor when the first frequency is applied is determined.

[0226] The specific implementation of determining the first operating parameter (such as the first frequency) can be found in the previous related introduction.

[0227] It is understood that this embodiment is an example of one implementation for determining the first operating parameter, and the other implementations for determining the first operating parameter mentioned above are also applicable to this embodiment.

[0228] S705 sets the target processor's operating parameters to the first operating parameters and uses the target processor to execute the inference of the target inference stage.

[0229] The first operating parameters may include a first frequency and a first voltage. Accordingly, the frequency and voltage of the target processor can be set as the first target frequency and the first voltage, respectively.

[0230] S706, during the inference process of the target processor executing the target inference stage, obtain the current second power supply capability information of the electronic device, the current temperature information of the target processor, and the performance and energy efficiency level of the electronic device.

[0231] The second power supply capability information is used to characterize at least one of the power supply type, available power, and power supply parameters of the electronic device.

[0232] For example, the performance equivalence level of an electronic device is used to characterize that the electronic device is in one of the following categories: high performance level, low energy consumption level, or performance-energy consumption balance level.

[0233] S707, determine the target operator currently being executed in the target inference phase.

[0234] S708, based on the temperature information, performance efficiency level, second power supply capability information, and the limiting operating parameters of the target operator on the target processor, determines the suitable second operating parameters for the target processor.

[0235] The second operating parameters may include a second frequency and a second voltage suitable for the target processor.

[0236] For example, in one possible scenario, based on the temperature information, performance efficiency level, the limiting operating parameters corresponding to the target operator (such as the limiting frequency corresponding to the target operator), and the second power supply capability information, a prediction model is used to predict the suitable second frequency for the target processor, and the suitable second voltage for the target processor is determined based on the second frequency, thus obtaining the suitable second frequency and second voltage for the target processor.

[0237] In another possible scenario, the highest permissible reference frequency for the target processor is determined based on the performance and energy efficiency level of the electronic device. Based on this, a parameter adjustment strategy can be determined by combining temperature information and second power supply capability information. Based on this parameter adjustment strategy, as well as the limiting frequency corresponding to the target operator and the reference frequency of the target processor, a second frequency is determined.

[0238] The parameter adjustment strategy can have several possibilities. For example, the parameter adjustment strategy can be a frequency reduction magnitude, and the frequency reduction magnitude corresponding to different reference adjustment frequencies is different.

[0239] For example, determine the maximum of the limiting frequency corresponding to the target operator and the reference frequency of the target processor, take the maximum as the base frequency, determine the frequency reduction range according to the determined parameter adjustment strategy, adjust the base frequency based on the frequency reduction range, and take the adjusted frequency as the second frequency.

[0240] In another possible scenario, if the performance efficiency level is high-performance (i.e., the electronic device is used in high-performance mode), then it is necessary to control the processor to run at a higher frequency as much as possible. Therefore, by combining temperature information and secondary power supply capability information, a suitable secondary frequency for the target processor can be determined. This secondary frequency should not exceed the limiting frequency corresponding to the target operator. For details, please refer to any of the implementation methods mentioned above.

[0241] If the performance efficiency level is low power level (that is, the electronic device is used in low power mode), then the electronic device needs to be kept in low power mode. At this time, the power supply capacity and temperature information of the electronic device can be ignored. The appropriate second frequency of the processor can be set to the minimum frequency between the reference frequency of the processor in this low power mode and the limit frequency corresponding to the target operator.

[0242] If the performance and energy efficiency level is a level that balances performance and energy efficiency (i.e., the usage mode of the electronic device is a performance and energy efficiency balance mode), then it is necessary to balance the performance and energy efficiency of the processor as much as possible. Therefore, temperature information and second power supply capability information can be combined to determine the suitable candidate frequency for the target processor, and the minimum frequency between the candidate frequency and the limiting frequency corresponding to the target operator is determined as the second frequency.

[0243] Of course, the above are just examples to illustrate a few situations. In practical applications, there may be other ways to determine the second frequency, and there are no restrictions on this.

[0244] S709, the operating parameters of the target processor are set to the second operating parameters so that the target processor continues to execute the inference of the target inference stage using the second operating parameters.

[0245] Of course, in practical applications, this application can continuously collect the actual performance and power consumption of the target processor during the target inference stage under different operating parameters, and continuously update and adjust the relevant strategies for operating parameters, which will not be elaborated further.

[0246] To facilitate understanding of the benefits of this application, an application scenario will be used as an example:

[0247] Taking the large language model as an example of the model of electronic device operation, as introduced above, the reasoning stage of the large language model includes the pre-filling stage and the decoding stage.

[0248] For ease of description, let's take the performance and energy efficiency level of the electronic device as an example, which represents the level that requires a balance between performance and energy efficiency (i.e., the usage mode is a balanced mode that balances performance and energy consumption). In this mode, the appropriate frequency for the electronic device can be determined by combining information such as the power supply capacity of the electronic device and the target inference stage performed by the electronic device.

[0249] During the execution of the target inference phase by the target processor, this application can determine the power supply capability information of the electronic device, the target inference phase executed by the electronic device, and the target operator executed in the target inference phase.

[0250] Based on the power supply capability information of electronic devices, the power supply status level of electronic devices is determined.

[0251] In cases where the power supply status is considered sufficient, and the target inference stage is either a pre-filling stage or a decoding stage, the target processor frequency can be increased as much as possible, provided it does not exceed the maximum allowed frequency (i.e., the limiting frequency) of the target operator on the target processor, in order to achieve high-efficiency operation of the target inference stage. For example... Figure 8 This diagram illustrates example frequency adjustments for the pre-filling and decoding stages of a large language model executed by the processor under different power supply states. Figure 8 As can be seen from curve 801, under sufficient power supply conditions, the processor frequency for both the pre-filling and decoding stages is relatively high.

[0252] If the power supply status is limited, and the target inference phase is a pre-filling phase, the current frequency of the target processor can be appropriately reduced, provided it does not exceed the maximum allowed frequency (i.e., the limiting frequency) of the target operator on that target processor. If the target inference phase is a decoding phase, the target processor frequency can be maintained as much as possible, provided it does not exceed the maximum allowed frequency (i.e., the limiting frequency) of the target operator on that target processor. Figure 8 As can be seen from curve 802, when the power supply is relatively limited, the processor executes the pre-filling stage of the large language model at a relatively low frequency, sacrificing a small amount of the delay time of the first word; and it will ensure the frequency of the processor executing the decoding stage in order to maintain the generation speed of each candidate word and ensure the smoothness of the model inference.

[0253] If the power supply status level is insufficient, the frequency of the target processor can be reduced, resulting in relatively lower frequencies for both the pre-filling and decoding stages. Figure 8 As shown in curve 803. For example, the frequency of the target processor is reduced to the set minimum operating frequency to ensure stable operation of the model inference and meet power consumption constraints.

[0254] Furthermore, this application also provides an electronic device in its embodiments. For example... Figure 9 As shown, it illustrates a schematic diagram of the composition structure of the electronic device, which includes at least a controller 901 and at least one processor 902, wherein different processors have different performance characteristics;

[0255] The controller 901 is configured to determine the target inference stage to be executed by the model, wherein the target inference stage is any inference stage in the model's inference process; determine the resource requirements of the target inference stage; determine a target processor from the at least one processor based on the resource requirements of the target inference stage and the performance characteristics of at least one candidate processor; and instruct the target processor to execute the inference of the target inference stage.

[0256] In response to the instruction of the controller, the processor 902 performs the inference of the target inference phase.

[0257] In this embodiment, the controller can be a microcontroller unit, CPU, or intelligent control chip that is independent of the processor, and there are no restrictions on this.

[0258] The processor can be any processor capable of running the model, such as the NPU or GPU mentioned earlier, which will not be elaborated further here.

[0259] Furthermore, the electronic device may also include a memory 903 for storing programs required for the controller to perform operations.

[0260] Understandably, the electronic device may also include a display unit 904 for displaying the inference results of the model, etc.

[0261] Of course, the electronic device can also have more than Figure 9 There are no restrictions on the number of more or fewer components.

[0262] Furthermore, such as Figure 10 The diagram shows another possible structural configuration of the electronic device of this application.

[0263] Depend on Figure 10 As can be seen, in addition to the controller 901 and at least one processor 902, the electronic device may also include a power supply module 905 and a power management integrated circuit (PMIC) module 906.

[0264] The power supply module 905 is used to supply power to electronic devices.

[0265] The controller is specifically configured to determine a target processor and suitable first operating parameters for the target processor from the at least one processor based on the resource requirements of the target inference stage and the performance characteristics of the at least one processor.

[0266] This power management integrated circuit module is used to set the processor's operating parameters to the first operating parameters based on the controller's instructions.

[0267] Furthermore, the electronic device includes a detection module 907 for obtaining current power supply capability information of the electronic device. This power supply capability information characterizes at least one of the electronic device's power type, available power, and power supply parameters. For example, the power supply capability information of the electronic device can be obtained before the processor executes the target inference phase and during the inference process of executing the target inference phase.

[0268] The controller is also used to determine a suitable second operating parameter for the target processor based on the power supply capability information currently determined by the detection module and the resource requirements of the target inference stage during the execution of the target inference stage by the target processor; and to control the power management integrated circuit module to set the operating parameter of the target processor to the second operating parameter.

[0269] Furthermore, the electronic device may also include at least one temperature sensor, which can be used to sense the temperature information of the processor, or the temperature information of the electronic device, etc.

[0270] For example, in one possible implementation, the temperature sensor is used to obtain the current temperature information of the target processor during the inference process of the target inference phase;

[0271] Furthermore, the controller is also used to determine at least one of the performance efficiency levels of the electronic device during the inference process of the target inference phase executed by the target processor.

[0272] Based on this, when determining the second operating parameters, the controller specifically determines the second operating parameters suitable for the target processor based on at least one of the temperature information and performance efficiency level, as well as the power supply capacity information and the resource requirements of the target inference stage.

[0273] This application also provides a computer program product, including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the model operation control methods provided in this application.

[0274] This application also provides a computer-readable storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the model operation control methods provided in this application.

[0275] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0276] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0277] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0278] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A model operation control method, comprising: Determine the target inference stage to be executed by the model, wherein the target inference stage is any inference stage in the model's inference process; Determine the resource requirements for the target reasoning phase; Based on the resource requirements of the target inference stage and the performance characteristics of at least one processor, a target processor is determined from the at least one processor, and different processors have different performance characteristics; The target processor is used to perform the reasoning in the target reasoning phase.

2. The model operation control method according to claim 1 further includes: In response to the target processor completing the inference of the target inference stage and the target inference stage not being the last inference stage of the model, the target processor is instructed to store the inference state data inferred in the target inference stage into a shared memory area; Wherein, the shared memory area is a memory area accessible to the at least one processor; The inference state data is the reference data required to execute the next inference stage in the model after the target inference stage.

3. The model operation control method according to claim 2, wherein the step of using the target processor to execute the inference of the target inference stage includes: If the target inference stage is not the first inference stage of the model, instruct the target processor to obtain the inference state data inferred from the previous inference stage of the target inference stage from the shared memory area; Based on the reasoning state data of the previous reasoning stage, the reasoning of the target reasoning stage is executed using the target processor.

4. The model operation control method according to claim 1, wherein determining the resource requirements of the target inference stage includes: Determine the resource requirement type for the target inference stage, wherein the resource requirement type is either computationally intensive or memory access intensive. The step of determining the target processor from the at least one processor based on the resource requirements of the target inference stage and the performance characteristics of the at least one processor includes: In response to the resource requirement type of the target inference stage being compute-intensive, a target processor whose computing performance meets the first set requirement is selected from the at least one processor; In response to the resource requirement type of the target inference stage being memory access intensive, a target processor whose power consumption performance meets the second set requirements is selected from at least one processor.

5. The model operation control method according to claim 1 further includes: In response to determining the target inference stage, the current resource status information of the electronic device is obtained, and the resource status information is used to characterize at least the current load status of each processor in the electronic device; The step of determining the target processor from the at least one processor based on the resource requirements of the target inference stage and the performance characteristics of the at least one processor includes: Based on the resource requirements of the target inference stage, the performance characteristics of at least one processor, and the resource status information, a target processor is determined from the at least one processor.

6. The model operation control method according to claim 1, wherein determining the target processor from the at least one processor based on the resource requirements of the target inference stage and the performance characteristics of at least one processor comprises: Based on the resource requirements of the target inference stage and the performance characteristics of at least one processor, a target processor and suitable first operating parameters for the target processor are determined from the at least one processor. The execution of the inference phase of the target inference using the target processor includes: The target processor is controlled to perform the inference of the target inference phase using the first operating parameters.

7. The model operation control method according to claim 6 further includes: During the inference process of the target inference stage executed by the target processor, the current power supply capability information of the electronic device is obtained. The power supply capability information is used to characterize at least one of the power supply type, available power, and power supply parameters of the electronic device. Based on the power supply capacity information and the resource requirements of the target inference stage, determine the second operating parameters suitable for the target processor; Set the operating parameters of the target processor to the second operating parameters.

8. The model operation control method according to claim 7, wherein the resource requirements of the target inference stage include: The target inference stage has restricted operating parameters for at least one operator on different processors, the restricted operating parameters being determined through testing, and the highest operating parameters allowed for the processor to run the operator; The step of determining suitable second operating parameters for the target processor based on the power supply capacity information and the resource requirements of the target inference stage includes: Determine the target operator currently being executed in the target inference phase; Based on the power supply capacity information and the limiting operating parameters of the target operator on the target processor, a second set of suitable operating parameters for the target processor is determined.

9. The model operation control method according to claim 7 further includes: During the inference process of the target processor executing the target inference stage, at least one of the current temperature information of the target processor and the performance energy efficiency level of the electronic device is obtained; The step of determining suitable second operating parameters for the target processor based on the power supply capacity information and the resource requirements of the target inference stage includes: Based on at least one of the temperature information and performance efficiency level, as well as the power supply capacity information and the resource requirements of the target inference stage, a second set of suitable operating parameters for the target processor is determined.

10. An electronic device, comprising: A controller and at least one processor, wherein different processors have different performance characteristics; The controller is configured to: determine the target inference stage to be executed by the model, wherein the target inference stage is any inference stage in the model's inference process; determine the resource requirements of the target inference stage; determine a target processor from the at least one processor based on the resource requirements of the target inference stage and the performance characteristics of at least one candidate processor; and instruct the target processor to execute the inference of the target inference stage. The processor is configured to perform inference in the target inference phase in response to an instruction from the controller.

Images