A method and system for multiple artificial intelligence model execution

Abstract

The present invention provides a method and system for executing multiple artificial intelligence (AI) models, wherein the method is used to optimize the parameter settings for compiling and executing the AI models. The AI models are used in an application scenario that runs on edge devices. The parameter settings include compilation settings and execution settings, which further include, for each AI model, hardware resource allocation and the order of executing the AI model in the application scenario, respectively. The parameter settings are adjusted using a genetic algorithm based on performance metrics to iteratively update the parameter settings in multiple iterations. In each iteration, the performance metrics are generated by one or more target edge devices executing the AI models according to the updated execution settings, and the AI models are compiled according to the updated compilation settings. At the end of the tuning, the optimized execution settings and the AI models compiled using the optimized compilation settings are sent to the edge devices to execute the AI models in the application scenario.

Description

Technical Field

[0001] Embodiments of the present invention relate to the optimization of configuration parameters for compiling and executing AI models. Background Technology

[0002] When an edge device runs multiple artificial intelligence (AI) applications simultaneously, the AI ​​models used by these applications may compete for limited system resources, such as central processing unit (CPU) cores, neural processing unit (NPU) and various memory devices. This competition can lead to performance degradation in AI applications, which may result in failure to meet expected performance requirements, such as power consumption, memory usage, and latency.

[0003] To mitigate potential performance degradation, application developers may rely on their experience with system resource allocation to meet performance requirements. However, the complexity of resource allocation increases dramatically when multiple AI applications are executed. There are countless possible combinations of resource allocation configuration settings, and only a limited set of settings can be experimented with within a limited timeframe. Furthermore, configuration tweaking involves trade-offs in performance metrics (e.g., increased latency as power consumption decreases), making it difficult for heuristic methods to find a configuration that balances the performance of multiple AI applications using AI models on edge devices.

[0004] Therefore, an efficient method is needed to determine resource allocation so that AI applications using AI models on edge devices can run simultaneously. This method not only needs to be efficient, but also needs to ensure that all AI applications can meet their performance requirements when running in parallel. Summary of the Invention

[0005] In one embodiment, a method is provided for optimizing the execution of an AI model used in an application scenario. The method includes initializing parameter settings for compiling and executing the AI ​​model. These parameter settings include compilation settings and execution settings, further including hardware resource allocation for at least each AI model and the order in which the AI ​​model is executed in the application scenario. The method also includes adjusting the parameter settings using a performance metric-based genetic algorithm to iteratively update the parameter settings to generate optimized compilation settings and optimized execution settings. In each iteration, the performance metric is generated by one or more target edge devices executing the AI ​​model according to the updated execution settings, and the AI ​​model is compiled according to the updated compilation settings. The method further includes sending the optimized execution settings and the AI ​​model compiled using the optimized compilation settings to multiple edge devices for execution of the AI ​​model in the application scenario.

[0006] In another embodiment, a system for optimizing the execution of an AI model used in an application scenario is provided. The system includes a processor and memory coupled to a processor. The processor is capable of performing the methods described above.

[0007] Other aspects and features will be understood by those skilled in the art from the following description of specific embodiments in conjunction with the accompanying drawings. Attached Figure Description

[0008] This invention is illustrated by way of example in the accompanying drawings, and not by way of limitation, wherein similar reference numerals denote similar elements. It should be noted that different references to "one" or "an embodiment" in this disclosure do not necessarily refer to the same embodiment, and such references imply at least one. Furthermore, when a particular feature, structure, or characteristic is described in relation to one embodiment, the submission is intended to enable those skilled in the art to implement such feature, structure, or characteristic in relation to other embodiments, whether explicitly described or not.

[0009] Figure 1 An example scenario according to one embodiment is shown, which includes an AI application with multiple features competing for resources on an edge device.

[0010] Figure 2 An optimizer for optimizing parameter settings is shown according to one embodiment.

[0011] Figure 3 An example of configuration parameters for an AI model according to one embodiment is shown.

[0012] Figure 4 It is a flowchart illustrating a method for generating optimized parameter settings using a genetic algorithm according to one embodiment.

[0013] Figure 5A and Figure 5B An example of the individual selection step in a genetic algorithm according to one embodiment is shown.

[0014] Figure 6 It is a flowchart illustrating a method for optimizing AI model execution according to one embodiment.

[0015] Figure 7 An example system for optimizing AI model execution is shown according to one embodiment. Detailed Implementation

[0016] Numerous specific details are set forth in the following description. However, it should be understood that embodiments of the invention can be carried out without these specific details. In other instances, well-known circuits, structures, and techniques are not shown in detail so as not to obscure the understanding of this description. However, those skilled in the art will understand that the invention can be carried out without these specific details. Those skilled in the art will be able to implement appropriate functionality without improper experimentation based on the included description.

[0017] This invention discloses an efficient method for optimizing compile-time and runtime parameter settings of AI models running on edge devices. These parameters, also known as configuration parameters, configure the AI ​​model at compile time and runtime to effectively utilize edge device resources. The method described in this invention can be applied to AI applications that use any number of concurrently running AI models on edge devices, including cases using only a single AI model. When multiple AI models are running concurrently, this method can be used to balance the competing demands of AI models on AI-specific hardware resources, such as Neural Processing Unit (NPU) cores and Tightly Coupled Memory (TCM). Given constraints, the parameter settings are iteratively adjusted to improve a set of performance metrics aimed at meeting given performance requirements. Non-limiting examples of performance metrics include power consumption, latency, memory footprint, etc. In one embodiment, optimization is performed using genetic algorithms, such as NSGA-II, SPEA2, MOEA / D, etc., which are known in the field of genetic algorithms.

[0018] In the following description, the term "AI feature" or its equivalent "feature" is used to denote the AI ​​functionality of an AI application, wherein the feature uses one or more AI models to perform operations. A "scenario" (also referred to as an "application scenario") includes one or more AI applications (also referred to as "applications"), each application including one or more features. In embodiments described in this invention, a scenario may include two or more concurrently executing applications and / or two or more concurrently executing features. The terms "concurrent" and "parallel" are used interchangeably. When multiple AI models are used in a scenario, the AI ​​models may run sequentially, in parallel, or partially sequentially and partially in parallel. The term "edge device" refers to an electronic device at a network endpoint, typically located at the user end. Non-limiting examples of edge devices include smartphones, autonomous vehicles, Internet of Things (IoT) devices, desktop / laptop computers, sensors, wearable devices, etc. The terms "compile-time parameter settings" and "compile settings" are used interchangeably, as are the terms "execution-time parameter settings" and "execution settings."

[0019] Figure 1An example scenario is shown, illustrating a situation where multiple features compete for hardware resources on edge device 100. In this example, the scenario includes an application—a camera application 110—that incorporates multiple features such as scene detection 120, capture noise reduction (NR) 130, face detection 140, etc. Features 120, 130, and 140 may each use one or more AI models to perform operations, such as AI models 125, 135, and 145, respectively. AI models 125, 135, and 145 compete for hardware resources 155 on edge device 100, such as TCM 160, Dynamic Random Access Memory (DRAM) 170, NPU cores 180, Central Processing Unit (CPU) cores 190, etc. It should be understood that edge device 100 may have alternative or additional hardware resources. TCM 160 is a small, fast memory located on-chip to provide low-latency memory access. NPU cores 180 provide deep learning capabilities and computer vision processing capabilities. In one embodiment, the NPU may include a deep learning accelerator (DLA) and a vision processing unit (VPU), both of which may have multiple cores. NPU core 180 may include both DLA cores and VPU cores. In some systems, the NPU is also referred to as an AI processing unit (APU). NPU core 180 and TCM 160 may be collectively referred to as "AI hardware" or "AI hardware resources".

[0020] In one embodiment, optimizer 150 calculates parameter settings for compiling and executing AI models 125, 135, and 145, which are then used in a scenario such as camera application 110. The parameter settings are optimized based on performance requirements such as power consumption, feature latency (e.g., latency for each feature 120, 130, and 140), memory footprint, etc. In this example, power consumption, feature latency, and memory footprint are measured by running camera application 110 on edge device 100. Optimizer 150 uses a genetic algorithm for optimization. The host computer can compile AI models 125, 135, and 145 based on the optimized compile-time parameter settings. Edge device 100 uses the optimized execution-time parameter settings to execute AI models 125, 135, and 145, as well as camera application 110.

[0021] It should be noted that the method for optimizing parameter settings described below is not limited to scenarios involving multiple concurrent executions of AI models. The optimization method can also be applied to scenarios where only one AI model is executed at a time on an edge device. For example, a given AI model can be compiled on a host computer (e.g., a server, personal computer, or similar device) with optimized compile-time parameter settings. The compiled AI model can then be executed on an edge device as part of an application (e.g., search, computer vision, etc.) with optimized execution-time parameter settings. For simplicity, the following embodiments use a scenario with multiple concurrent executions of AI models for description and illustration.

[0022] Figure 2 This is a block diagram illustrating optimizer 150 according to one embodiment. Optimizer 150 optimizes parameter settings for a given scene, including one or more features and multiple AI models used by one or more features. Parameter settings include compilation settings and execution settings. Optimizer 150 includes a tuner 210 for tuning the parameter settings for compiling and executing features and AI models. Tuner 210 receives AI models and tuning configurations, including but not limited to specifications (e.g., performance requirements) and constraints (e.g., system constraints, such as the number of available NPU cores, the available TCM size, etc., and dependency constraints between AI models), and outputs a configuration containing parameter settings. Dependency constraints can be specified per feature. For a scene that generates image frames as output, performance requirements can include the number of frames per second (FPS) required for the images. FPS constraints can be specified per feature. Optimizer 150 also includes an evaluator 220 that receives the configuration (containing the given scene) and evaluates the performance of the given scene on a target edge device to output a set of performance metrics, such as power consumption, feature latency, memory footprint, etc. Tuner 210 then updates the parameter settings based on the performance metrics. The operations of tuner 210 and evaluator 220 are repeated in multiple iterations until a termination condition is met. In one embodiment, the operation of tuner 210 can be performed by a host computer. Evaluator 220 also includes a compilation module 230 to compile the AI ​​model according to the compilation settings in each iteration. Evaluator 220 also includes an execution module 240 to run a given scene with the compiled AI model according to the execution settings in each iteration. The operation of compilation module 230 can be performed by a host computer or an edge device. The operation of execution module 240 can be performed by a target edge device. In one embodiment, the target edge device has the same AI execution hardware as the edge device where the AI ​​model will be deployed. In one embodiment, multiple target edge devices can run a given scene in parallel to generate performance metrics, such as... Figure 4 The examples in the document are described in further detail.

[0023] Figure 3 This section illustrates an example of configuring AI model parameters according to one embodiment. See also... Figure 2 Scenario 300 can be provided as input to optimizer 150 to determine the parameter settings for optimization. In this example, scenario 300 includes two features, each using multiple AI models (e.g., feature 1 uses model 1, model 2, and model 3, and feature 2 uses model 4 and model 5). Each AI model has the same configuration parameters for optimization, such as three compile-time parameters CP1, CP2, and CP3 and three execution-time parameters EP1, EP2, and EP3. Each feature also includes a model execution order parameter. It should be understood that... Figure 3 This is a non-restrictive example, simplified for ease of explanation. For example, CP1, CP2, and CP3 could represent the TCM size, the number of NPU cores, and the slicing and fusion strategy, respectively. EP1, EP2, and EP3 could represent the NPU Operational Performance Points (OPPs), the number of threads, and the scheduling strategy, respectively. The slicing and fusion strategy for the AI ​​model could include dividing some computations in the AI ​​model into smaller slices and combining some computations to improve processor and memory utilization efficiency. The scheduling strategy for the AI ​​model could include rules and define the priority for scheduling AI model computations. The model execution order could define which AI models should be executed sequentially and which should be executed concurrently (i.e., in parallel) for each feature. For example, Models 1, 2, and 3 for Feature 1 should be executed sequentially, while Models 4 and 5 for Feature 2 should be executed in parallel.

[0024] In some embodiments, the execution order of models can be described by a directed graph, where each node represents an AI model and the directed edge between any two nodes represents the execution order of the two nodes. A directed graph can be used to describe the execution order of any number of AI models, some of which can be executed in parallel and some sequentially.

[0025] See Figure 1 and Figure 2 The tuner 210 initializes the parameter settings under the constraints provided by the input. Using a genetic algorithm, the optimizer 150 iteratively updates the parameter settings guided by performance metrics.

[0026] Figure 4 This is a flowchart illustrating a method 400 for generating optimized parameter settings using a genetic algorithm, according to one embodiment. (Reference) Figure 2The method 400 begins at step 410 when the tuner 210 receives input containing the AI ​​models, specifications, and constraints used in a given scenario. In step 420, the tuner 210 initializes an individual population, each represented by a set of parameters. The initial population can be generated randomly or based on prior knowledge. As a simplified example, the given scenario includes a specific function using two AI models, M1 and M2, which can execute independently. Optimization objectives according to the specifications include a target function latency of 150 milliseconds (ms) and a target power consumption of 200 milliamps (mA). Constraints include the number of NPU cores selected from the following: {1, 2, 3, 4}, the TCM size selected from the following: {0, 1, …, 7} megabytes (MB), the NPU OPP selected from the following: {0, 1, …, 10}, and the model execution order selected from the following: {M1 and M2 in parallel, M1 before M2, M2 before M1}. An individual in the initial population can be represented as follows. M1: NPU cores = 3, TCM size = 2MB, NPUOPP = 6; M2: NPU cores = 1, TCM size = 5MB, NPU OPP = 8; M1 and M2 execute in parallel. A simplified representation could be: (3, 2, 6; 1, 5, 8; M1 and M2 in parallel).

[0027] In step 430, evaluator 220 evaluates the population by compiling an AI model, and then executes each individual on the target edge devices. The AI ​​model is recompiled for each selected individual in the genetic algorithm in each iteration (i.e., generation). The evaluation output is a performance metric for each individual in a given scenario, including functional latency and power consumption. In one embodiment, multiple target edge devices with the same hardware model can be executed in parallel to evaluate different individuals, with each target edge device executing one or more individuals, thereby accelerating the evaluation process. The performance of the same individuals executed on these edge devices is substantially the same, or at least their performance differences do not exceed the variations caused by adjusting compilation and / or execution parameter settings. After evaluator 220 generates the performance metrics for the entire population, in step 440, tuner 210 selects a subset of individuals from the population based on the performance metrics. According to the NSGA-II genetic algorithm, the individual selection in step 440 includes two sub-steps. The first sub-step is to rank the individuals in the population into multiple frontiers based on non-dominated sorting. The second sub-step is to calculate the crowding distance. Assuming the population consists of 9 individual AIs, an example of the performance metrics for each individual is as follows: Figure 5A As shown. Figure 5BThree fronts formed by nine individuals are shown. Individuals in front 1 have a higher selection priority than those in front 2, which in turn have a higher selection priority than those in front 3. Within the same front, individuals with higher crowding distances have higher selection priority. A description of the front and crowding distance calculations can be found in the article "A fast and elitist multi-objective genetic algorithm: NSGA-II," by K. Deb et al., published in IEEE Transactions on Evolutionary Computation, vol. 6, no. 2, pp. 182-197, April 2002.

[0028] After selecting a subset of individuals in step 450, optimizer 150 (e.g., tuner 210) generates offspring for those individuals selected in step 440. Offspring can be generated through two sub-steps. The first sub-step for offspring generation is crossover. Assume individual G is represented as (3, 2, 6; 1, 5, 8; M1 and M2 in parallel), and individual C is represented as (2, 3, 4; 4, 1, 9; M1 before M2). After crossover, two new individuals G' and C' are generated: G': (3, 3, 4; 1, 1, 9; M1 and M2 in parallel), C': (2, 2, 6; 4, 5, 8; M1 before M2). The second sub-step for offspring generation is mutation. After crossover, one or more genes (i.e., parameters) of each individual may be randomly mutated. For example, the first parameter of an individual G' may vary as (4, 3, 4; 1, 1, 9; M1 and M2 in parallel).

[0029] After offspring are generated, optimizer 210 updates the population in step 460 by combining the individuals selected in step 440 with the offspring generated in step 450. Steps 430-460 are repeated until a termination condition is met, such as reaching a predetermined number of iterations or the performance metrics meet the specifications. In step 470, optimizer 150 outputs optimized parameter settings for the two AI models M1 and M2 used in the given scenario.

[0030] Figure 6 This is a flowchart illustrating a method 600 for optimizing AI model execution using a genetic algorithm, according to one embodiment. Method 600 can be executed by a host computer system, such as... Figure 7 System 700. An example of the target edge device mentioned in method 600 could be... Figure 1 100 edge devices.

[0031] Reference Figure 6Method 600 begins at step 610, where the computer system initializes parameter settings for compiling and executing AI models used in the application scenario. These parameter settings include compilation settings and execution settings, further including at least the hardware resource allocation for each AI model and the order in which the AI ​​models are executed in the application scenario. At step 620, the computer system uses a performance metric-based genetic algorithm to adjust these parameter settings iteratively, updating them to generate optimized compilation and execution settings. In each iteration, the performance metric is generated by one or more target edge devices executing the AI ​​model according to the updated execution settings, and the AI ​​model is compiled according to the updated compilation settings. At step 630, the computer system sends the optimized execution settings and the AI ​​model compiled using the optimized compilation settings to multiple edge devices for execution in the application scenario. In one embodiment, once optimization is complete, the host computer can send the optimized parameter settings to a storage location for users to download to their edge devices. Therefore, the term "send" in step 630 can include sending directly or indirectly to the user's edge device.

[0032] In one embodiment, hardware resource allocation includes the allocation of one or more of the following: TCM and NPU cores. In one embodiment, the AI ​​model is recompiled for each selected individual of the genetic algorithm. In one embodiment, the parameter settings include compile-time parameters and execution-time parameters. Compile-time parameters include one or more of the following: the memory size allocated to each AI model, the number of processor cores allocated to each AI model, and the slicing and fusion strategy for each AI model. Execution-time parameters include one or more of the following: the OPP for the NPU used to execute each AI model, the number of threads for each AI model, and the scheduling strategy. In one embodiment, the parameter settings include a directed graph representing the execution order of the AI ​​models. In one embodiment, the two or more AI models are executed concurrently on the target edge device in the application scenario.

[0033] In one embodiment, the computer system may receive a first tuning configuration indicating the FPS required from image frames generated from executing the scene, and a second tuning configuration indicating dependency constraints between the AI ​​models. The computer system adjusts the parameter settings according to the first and second tuning configurations. In one embodiment, multiple target edge devices may execute in parallel during the tuning, with each target edge device executing one or more individuals of the genetic algorithm to generate the performance metric. In one embodiment, each individual in the genetic algorithm is represented by the compile-time and execution-time parameters of all the AI ​​models and the execution order of the AI ​​models.

[0034] Figure 7An example of a system 700 for optimizing AI model execution according to one embodiment is shown. System 700 can be an execution Figure 6 Method 600 refers to a host computer. In this example, system 700 includes multiple processors 710, such as a central processing unit (CPU), a graphics processing unit (GPU), a multimedia processor, a digital signal processor, and other general-purpose and / or special-purpose processing circuitry. Each CPU and GPU may include multiple cores.

[0035] System 700 also includes memory 720. Memory 720 may include a dynamic random access memory (DRAM) device, a static RAM (SRAM) device, a flash memory device, and / or other volatile or non-volatile storage devices. In one embodiment, memory 720 stores instructions executable by processor 710 to adjust parameter settings according to genetic algorithm 770. In another embodiment, memory 720 may store... Figure 2 The tuner 210 in the middle. Although in Figure 7 Memory 720 is shown as a block, but it can be understood that memory 720 may include multiple memory devices located in multiple memory hierarchies.

[0036] System 700 may also include I / O circuitry 730 for receiving inputs and sending outputs. In one embodiment, system 700 may further include a network interface 750 for accessing wired and / or wireless networks. In some embodiments, system 700 may access external AI models via network interface 750. In one embodiment, system 700 may include a display 760 for displaying output generated by application 740. It is understood that system 700 is for illustrative purposes only; other hardware and software components are not shown.

[0037] Figure 4 and Figure 6 The flowchart operation has been referenced. Figure 1 and Figure 7 Exemplary embodiments have been described. However, it should be understood that... Figure 4 and Figure 6 The flowchart operations can be performed by other embodiments of the present invention, and are not limited to those described herein. Figure 1 and Figure 7 The embodiments, and Figure 1 and Figure 7 The embodiments may perform operations different from those discussed in the reference flowchart. It is understood that... Figure 4 and Figure 6 The order of operations shown in the flowchart is a non-limiting example. Alternative embodiments may perform operations in a different order, combine certain operations, overlap certain operations, etc.

[0038] Various functional components or modules are described in this invention. As those skilled in the art will understand, functional modules are preferably implemented by circuits (whether dedicated circuits or general-purpose circuits operating under the control of one or more processors and coded instructions), which typically include transistors configured to operate as the functional and operational control circuitry described in this invention.

[0039] While the invention has been described with reference to several embodiments, those skilled in the art will recognize that the invention is not limited to the described embodiments and that modifications and alterations can be made within the spirit and scope of the appended claims. Therefore, the description should be considered illustrative rather than restrictive.

Claims

1. A method for optimizing the execution of multiple artificial intelligence (AI) models used in an application scenario, characterized in that, include: Initialize parameter settings for compiling and executing the AI ​​model, the parameter settings including compilation settings and execution settings, which further include at least the hardware resource allocation for each AI model and the execution order of the AI ​​models in the application scenario; The parameter settings are adjusted using a performance metric-based genetic algorithm to iteratively update the parameter settings to generate optimized compilation settings and optimized execution settings, wherein in each iteration, the performance metrics are generated by one or more target edge devices executing the AI ​​model according to the updated execution settings, and the AI ​​model is compiled according to the updated compilation settings; as well as The optimized execution settings and the AI ​​model compiled using the optimized compile-time settings are sent to multiple edge devices to execute the AI ​​model in the application scenario.

2. The method as described in claim 1, characterized in that, The hardware resource allocation includes allocating one or more of the following: tightly coupled memory and neural processing unit cores.

3. The method as described in claim 1, characterized in that, The AI ​​model is a recompilation of multiple selected individuals from the genetic algorithm.

4. The method as described in claim 1, characterized in that, The parameter settings include the settings of compile-time parameters, and the compile-time parameters include one or more of the following: the memory size allocated to each AI model, the number of processor cores allocated to each AI model, and the slicing and fusion strategy for each AI model.

5. The method as described in claim 1, characterized in that, The parameter settings include the settings of execution parameters, and the execution parameters include one or more of the following: the operating performance point of the neural processing unit for executing each AI model, the number of threads for each AI model, and the scheduling strategy.

6. The method as described in claim 1, characterized in that, The parameter settings include a directed graph representing the execution order of the AI ​​model.

7. The method as described in claim 1, characterized in that, The two or more AI models are executed concurrently on the target edge device in the application scenario.

8. The method as described in claim 1, characterized in that, The plurality of target edge devices execute in parallel during the adjustment of the parameter settings, and each target edge device executes one or more individuals of the genetic algorithm to generate the performance metrics.

9. The method as described in claim 1, characterized in that, The method further includes: Receive a first tuning configuration indicating the number of frames per second required from the image frames generated from the application scenario, and a second tuning configuration indicating the dependency constraints between the AI ​​models; and Adjust the parameter settings according to the first tuning configuration and the second tuning configuration.

10. The method as described in claim 1, characterized in that, Each individual in the genetic algorithm is represented by the compile-time parameters and execution-time parameters of all the AI ​​models and the execution order of the AI ​​models.

11. A system for optimizing the execution of multiple artificial intelligence (AI) models used in an application scenario, characterized in that, include: Multiple processors; as well as Memory coupled to the plurality of processors, wherein one or more of the processors are capable of: Initialize parameter settings for compiling and executing the AI ​​model, the parameter settings including compilation settings and execution settings, which further include at least the hardware resource allocation for each AI model and the execution order of the AI ​​models in the application scenario; The parameter settings are adjusted using a performance metric-based genetic algorithm to iteratively update the parameter settings to generate optimized compilation settings and optimized execution settings, wherein in each iteration, the performance metrics are generated by one or more target edge devices executing the AI ​​model according to the updated execution settings, and the AI ​​model is compiled according to the updated compilation settings; as well as The optimized execution settings and the AI ​​model compiled using the optimized compile-time settings are sent to multiple edge devices to execute the AI ​​model in the application scenario.

12. The system as claimed in claim 11, characterized in that, The hardware resource allocation includes allocating one or more of the following: tightly coupled memory and neural processing unit cores.

13. The system as described in claim 11, characterized in that, The AI ​​model is a recompilation of multiple selected individuals from the genetic algorithm.

14. The system as claimed in claim 11, characterized in that, The parameter settings include the settings of compile-time parameters, and the compile-time parameters include one or more of the following: the memory size allocated to each AI model, the number of processor cores allocated to each AI model, and the slicing and fusion strategy for each AI model.

15. The system as claimed in claim 11, characterized in that, The parameter settings include the settings of execution parameters, and the execution parameters include one or more of the following: the performance point of the neural processing unit used to execute each AI model, the number of threads for each AI model, and the scheduling strategy.

16. The system as claimed in claim 11, characterized in that, The parameter settings include a directed graph representing the order in which the AI ​​models are executed.

17. The system as claimed in claim 11, characterized in that, Two or more AI models are executed concurrently on the target edge device in the application scenario.

18. The system as claimed in claim 11, characterized in that, Multiple target edge devices execute in parallel during the adjustment of the parameter settings, with each target edge device executing one or more genetic algorithm individuals to generate the performance metrics.

19. The system as claimed in claim 11, characterized in that, The one or more processors further operate to: Receive a first tuning configuration indicating the number of frames per second required from the image frames generated from the application scenario, and a second tuning configuration indicating the dependency constraints between AI models; and tune the parameter settings according to the first tuning configuration and the second tuning configuration.

20. The system as claimed in claim 11, characterized in that, Each individual in the genetic algorithm is represented by the compile-time and execution-time parameters of all AI models and the order in which the AI ​​models are executed.

Images