AI multi-model inference optimization method, inference device, readable storage medium, product

Abstract

The present application belongs to the technical field of AI chips, and particularly relates to an AI multi-model inference optimization method, an inference device, a readable storage medium and a product. The present application can automatically process each deep learning model in an AI service stream according to SRAM capacity and prior knowledge or an established model knowledge base to obtain a sub-model suitable for being solidified to the SRAM, and can run the AI service stream in a loop for the solution space corresponding to the sub-model and automatically perform a grid search to obtain a model combination meeting a performance target, wherein the corresponding sub-model in the model combination is a model directly solidified to the SRAM. The present application can achieve optimal chip throughput with a small SRAM space and a low DDR transmission bandwidth, and can reduce the power consumption and cost of the chip, thereby solving the technical problems of the prior art, such as single-model solidification and manual setting of solidified model parameters.

Description

Technical Field

[0001] This invention belongs to the field of AI chip technology, specifically relating to an AI multi-model inference optimization method, inference device, readable storage medium, and product. Background Technology

[0002] As AI chips become more powerful and neural network models become deeper, AI processors need to read a large amount of weight data for calculation when performing inference. This large data throughput will undoubtedly reduce the computing power of AI chips.

[0003] To reduce chip power consumption and cost, researchers have conducted in-depth research on neural network weight storage. Currently, the mainstream storage solutions include high-speed DDR storage (Double Data Rate SDRAM) combined with cache, or the use of ultra-large capacity SRAM (Static Random-Access Memory). Further solutions include ping-pong cache.

[0004] For storage and retrieval solutions combining high-speed DDR memory with L2 cache (secondary cache), such as... Figure 1 As shown, the weights of each layer of the neural network are stored in DDR memory. When the processor is calculating, it loads the weights to be processed into the cache through the high-speed DDR interface. However, the stronger the processor's computing power, the larger the DDR memory transmission bandwidth required by this scheme, which is not conducive to reducing the chip's power consumption and cost. In addition, it also has the drawbacks of large storage space occupation or difficulty in achieving optimal chip throughput.

[0005] For schemes that store model weights in a specified region of SRAM, such as Figure 2 As shown, SRAM is divided into two storage areas: static memory and circular cache. Static memory is used to store layers in the neural network that have underflowed, while the circular cache is used to prefetch and store layers in the neural network that have not underflowed and image input.

[0006] In this structure, the values ​​of neural network layers experiencing cache underflow are stored in static memory, which helps resolve the issue of circular cache underflow. When no layers in the neural network experience underflow, the maximum value of the accumulated length of the corresponding weights is set as the maximum storage space of the circular cache, thus helping to resolve the issue of circular cache overflow. Furthermore, it achieves optimal chip throughput with less SRAM space and lower DDR memory transfer bandwidth, thereby reducing chip power consumption and cost.

[0007] However, the main problem with this solution is:

[0008] 1. Parameters must be set manually, and developers need to have a good understanding of the hardware characteristics;

[0009] 2. Existing solutions are all based on a single model. For some complex multi-model business flows, the solution space for configuration parameters is too large, and it is difficult to achieve the optimal solution by manually setting parameters for optimization. Summary of the Invention

[0010] This invention provides an AI multi-model inference optimization method, inference device, readable storage medium, and product. It can automatically obtain the optimized combination of models stored in SRAM for multi-model business flows, solving the problems of storing model weights in a designated area of ​​SRAM. This invention requires less human intervention and improves the end-to-end inference performance of AI business flows.

[0011] The technical solution adopted in this invention is as follows:

[0012] An AI multi-model inference optimization method, the steps of which are as follows:

[0013] S1. Based on the business objectives to be achieved, determine the AI ​​business flow and obtain the deep learning models used in the AI ​​business flow. The number of deep learning models is Q, where Q≥2.

[0014] Once the AI ​​business flow is determined, the deep learning models used are also determined, as is the order in which they are executed. The business objectives are set according to actual needs, and the corresponding AI business flow could be a video structuring business flow, a speech recognition business flow, or an image optimization processing business flow, etc. Moreover, different AI business flows will use different deep learning models.

[0015] S2 loads the unoptimized deep learning models onto the AI ​​chip to obtain the runtime percentage and frequency of each deep learning model after the AI ​​chip runs the AI ​​business flow; the runtime percentage and frequency of each deep learning model are obtained by the AI ​​chip's built-in profiling tool when the AI ​​chip uses business data to perform AI business flow inference.

[0016] S3. Initially eliminate deep learning models with short running time and low frequency to obtain M deep learning models after initial elimination, where M < Q.

[0017] The initial elimination criteria are: the relationship between the running time t of the deep learning model and the total running time T satisfies t≤cT, 0<c<0.1; and the frequency f of the deep learning model ≤D, D∈[0,3].

[0018] Models that take a short time and are used infrequently are excluded from optimization to reduce the computational cost of solution space search. Generally, models that account for 90% of the total optimization time are selected.

[0019] S4, select from M deep learning models to obtain a set of models that can be embedded in SRAM, and the set of models contains N optimizable sub-models, where N < M;

[0020] There are two ways to filter:

[0021] One approach is to filter based on the relationship between the size of the deep learning model and the total capacity of SRAM, as well as prior knowledge, and then process each deep learning model in the specific AI business flow using prior knowledge.

[0022] The specific screening steps are as follows:

[0023] Compare the relationship between the current deep learning model size 'a' and the total SRAM capacity 'b'.

[0024] If a > b, then the current deep learning model is not suitable for fixation;

[0025] If b > a ≥ Kb, K ∈ [0.3, 0.5], then based on past experience, we can determine whether the current deep learning model can be split. If the current deep learning model cannot be split, then it is discarded.

[0026] If the current deep learning model can be decomposed, then based on prior knowledge, two sub-deep learning models are obtained, and the sub-deep learning models of different sizes are taken as optimizable sub-models.

[0027] If a < Kb, K ∈ [0.3, 0.5], then the current deep learning model can be fixed as an optimizable sub-model;

[0028] After each deep learning model is selected according to the screening method, a set of models that can be embedded in SRAM is obtained. The model set contains N optimizable sub-models, where N < M.

[0029] Another approach involves filtering based on the relationship between the size of the deep learning model and the total SRAM capacity, as well as the model knowledge base. Before determining the AI ​​workflow, a model knowledge base is established, storing the processing rules for each deep learning model. Each deep learning model's processing rule is based on prior knowledge to determine whether the deep learning model is decomposeable and the sub-models that can be decomposed. The number of deep learning models in the model knowledge base is greater than or equal to the number of deep learning models used in a given AI workflow. The deep learning models in the model knowledge base are obtained by first processing all deep learning models that might be used in each AI workflow to obtain the processing rules for each model. In a specific AI workflow, the processing rules for the deep learning models used are directly obtained from the model knowledge base.

[0030] The specific screening steps are as follows:

[0031] Compare the relationship between the current deep learning model size 'a' and the total SRAM capacity 'b'.

[0032] If a > b, then the current deep learning model is not suitable for fixation;

[0033] If b > a ≥ Kb, K ∈ [0.3, 0.5], then the model is processed according to the processing rules of the current deep learning model in the model knowledge base. If the current deep learning model cannot be split, it is removed.

[0034] If the current deep learning model can be split, then the two sub-deep learning models are obtained directly, and the sub-deep learning models of different sizes are taken as optimizable sub-models.

[0035] If a < Kb, K ∈ [0.3, 0.5], then the current deep learning model can be fixed as an optimizable sub-model;

[0036] After each deep learning model is selected according to the screening method, a set of models that can be embedded in SRAM is obtained. The model set contains N optimizable sub-models, where N < M.

[0037] S5, obtain the solution space of N optimizable sub-models. The solution space is a summary of the performance parameters obtained by running business data on the AI ​​chip for each optimization combination of the N optimizable sub-models.

[0038] On the AI ​​chip, for each optimization combination of N optimizable sub-models, the business flow is run using business data to obtain the performance parameters corresponding to each optimization combination, and the solution space of the N optimizable sub-models is obtained by summarizing them; the performance parameters include, but are not limited to, running time, memory usage, and power consumption, which are set according to actual conditions.

[0039] S6. Based on the required performance objective, a grid search is performed on the solution space. The optimized combination corresponding to the solution that meets the performance objective is the final optimized combination. The corresponding optimizable sub-models in the final optimized combination are stored in SRAM. The performance objective is the minimum value of a certain performance parameter, which can be set according to actual needs, such as minimum running time, minimum memory usage, or minimum power consumption.

[0040] The present invention also provides an inference device, including a processor, an AI chip, and a memory; the memory stores instructions executable by the processor, which, when executed by the processor, enable the processor to perform the steps of any of the AI ​​multi-model inference optimization methods described above; the AI ​​chip is used to run an AI business flow using a business flow, providing the processor with the required deep learning model's runtime percentage and frequency, and performance parameters of the optimized combination.

[0041] The present invention also provides a readable storage medium storing a program for implementing an AI multi-model inference optimization method, wherein the program for implementing the AI ​​multi-model inference optimization method is executed by a processor to implement the steps of the AI ​​multi-model inference optimization method as described above.

[0042] The present invention also provides a product, which is a computer program product, including a computer program, which is executed by a processor to implement the steps of the AI ​​multi-model inference optimization method described above.

[0043] This invention addresses the challenges of AI workflows with multiple deep learning models. It automatically processes each deep learning model in the workflow based on SRAM capacity and prior knowledge or an established model knowledge base to obtain suitable sub-models for SRAM. For the solution space corresponding to each sub-model, the AI ​​workflow is iteratively run, and a grid search is automatically performed to obtain a model combination that meets performance targets. The sub-models in this model combination are the models directly embedded in SRAM. This invention achieves optimal chip throughput with a smaller SRAM space and lower DDR storage bandwidth, reducing chip power consumption and cost. It also solves the existing technical problems of only being able to embed a single model and requiring manual setting of embedded model parameters. Attached Figure Description

[0044] Figure 1 This is a schematic diagram of a storage and retrieval scheme that combines high-speed DDR memory with L2 cache in existing technologies.

[0045] Figure 2 This is a schematic diagram of a scheme in the prior art that stores model weights in a designated area of ​​SRAM.

[0046] Figure 3 This is a flowchart illustrating the principle of the present invention.

[0047] Figure 4 This is a flowchart of the video structuring service flow of the present invention. Detailed Implementation

[0048] Example 1

[0049] An AI multi-model inference optimization method, such as Figure 3 As shown, the steps are as follows:

[0050] S1. Based on the business objectives to be achieved, determine the AI ​​business flow and obtain the deep learning models used in the AI ​​business flow. The number of deep learning models is Q, where Q≥2.

[0051] Once the AI ​​business flow is determined, the deep learning models used are also determined, as is the order in which they are executed. The business objectives are set according to actual needs, and the corresponding AI business flow could be a video structuring business flow, a speech recognition business flow, or an image optimization processing business flow, etc. Moreover, different AI business flows will use different deep learning models.

[0052] S2, load the unoptimized deep learning models onto the AI ​​chip, and obtain the running time percentage and frequency of each deep learning model after the AI ​​chip runs the AI ​​business flow; the running time percentage and frequency of each deep learning model are obtained by the AI ​​chip's built-in profiling tool when the AI ​​chip uses business data to perform AI business flow inference;

[0053] S3. Initially eliminate deep learning models with short running time and low frequency to obtain M deep learning models after initial elimination, where M < Q.

[0054] The initial elimination criteria are: the relationship between the running time t of the deep learning model and the total running time T satisfies t≤cT, 0<c<0.1; and the frequency f of the deep learning model ≤D, D∈[0,3].

[0055] Models that take a short time and are used infrequently are excluded from optimization to reduce the computational cost of solution space search. Generally, models that account for 90% of the total optimization time are selected.

[0056] S4, select from M deep learning models to obtain a set of models that can be embedded in SRAM, and the set of models contains N optimizable sub-models, where N < M;

[0057] There are two ways to filter:

[0058] One approach is to filter based on the relationship between the size of the deep learning model and the total capacity of SRAM, as well as prior knowledge, and then process each deep learning model in the specific AI business flow using prior knowledge.

[0059] The specific screening steps are as follows:

[0060] Compare the relationship between the current deep learning model size 'a' and the total SRAM capacity 'b'.

[0061] If a > b, then the current deep learning model is not suitable for fixation;

[0062] If b > a ≥ Kb, K ∈ [0.3 0.5], then based on past experience, we can determine whether the current deep learning model can be split. If the current deep learning model cannot be split, then it is removed.

[0063] If the current deep learning model can be split, then based on prior knowledge, two sub-deep learning models are obtained, and the sub-deep learning models of different sizes are taken as optimizable sub-models.

[0064] If a < Kb, K ∈ [0.3 0.5], then the current deep learning model can be fixed as an optimizable sub-model;

[0065] After each deep learning model is selected according to the screening method, a set of models that can be embedded in SRAM is obtained. The model set contains N optimizable sub-models, where N < M.

[0066] Another approach involves filtering based on the relationship between the size of the deep learning model and the total SRAM capacity, as well as the model knowledge base. Before determining the AI ​​workflow, a model knowledge base is established, storing the processing rules for each deep learning model. Each deep learning model's processing rule is based on prior knowledge to determine whether the deep learning model is decomposeable and the sub-models that can be decomposed. The number of deep learning models in the model knowledge base is greater than or equal to the number of deep learning models used in a given AI workflow. The deep learning models in the model knowledge base are obtained by first processing all deep learning models that might be used in each AI workflow to obtain the processing rules for each model. In a specific AI workflow, the processing rules for the deep learning models used are directly obtained from the model knowledge base.

[0067] The specific screening steps are as follows:

[0068] Compare the relationship between the current deep learning model size 'a' and the total SRAM capacity 'b'.

[0069] If a > b, then the current deep learning model is not suitable for fixation.

[0070] If b > a ≥ Kb, K ∈ [0.3, 0.5], then the model is processed according to the processing rules of the current deep learning model in the model knowledge base. If the current deep learning model cannot be split, it is removed.

[0071] If the current deep learning model can be split, then the two sub-deep learning models are obtained directly, and the sub-deep learning models of different sizes are taken as optimizable sub-models.

[0072] If a < Kb, K ∈ [0.3, 0.5], then the current deep learning model can be fixed as an optimizable sub-model;

[0073] After each deep learning model is selected according to the screening method, a set of models that can be embedded in SRAM is obtained. The model set contains N optimizable sub-models, where N < M.

[0074] S5, obtain the solution space of N optimizable sub-models. The solution space is a summary of the performance parameters obtained by running business data on the AI ​​chip for each optimization combination of the N optimizable sub-models.

[0075] On the AI ​​chip, for each optimization combination of N optimizable sub-models, the business flow is run using business data to obtain the performance parameters corresponding to each optimization combination, and the solution space of the N optimizable sub-models is obtained by summarizing them; the performance parameters include, but are not limited to, running time, memory usage, and power consumption, which are set according to actual conditions.

[0076] S6. Based on the required performance target, a grid search is performed on the solution space. The optimized combination corresponding to the solution that meets the performance target is the final optimized combination. The corresponding optimizable sub-models in the final optimized combination are stored in SRAM. At the same time, the hardware parameters of the final optimized combination are saved.

[0077] The performance target is the minimum value of a certain performance parameter, which can be set according to actual needs. It can be the minimum running time, the minimum memory usage, or the minimum power consumption.

[0078] The present invention will be described below using video structured service flow as an example.

[0079] like Figure 4 The diagram shown illustrates the principle of video structuring, a common application in security AI. It enables four functions. The explanation of the method can be based on any one, two, three, or all four functions implemented simultaneously. This study focuses on the business flow implementing vehicle structuring and non-motorized vehicle structuring. This flow involves approximately 14 deep learning models, specifically detection and classification models. One is a primary detection model with 640 inputs using YOLOv5S (model size approximately 28MB). Figure 4 Object detection in YOLOv5S (model size approximately 10M) with two secondary detections and 192 inputs. Figure 4 Secondary vehicle and non-motorized vehicle inspections; YOLOv5S is an abbreviation for deep learning detection network; 11 attribute recognition ResNet18 models (each model is approximately 1MB in size; ResNet18 is a type of deep learning residual network). Figure 4 The system includes 11 features such as vehicle type recognition, license plate recognition, driver attribute recognition, non-motorized vehicle recognition, clothing color recognition, and helmet recognition. Figure 4 In this context, color space conversion is achieved using the YUV2RGB function to convert between YUV and RGB color spaces.

[0080] According to the optimization method provided by this invention, the entire optimization process is roughly as follows:

[0081] 1. First, load the 14 unoptimized models onto the chip and use the dataset for business flow inference;

[0082] 2. Using the chip's built-in profiling tool, the runtime percentage and frequency of each model were determined. The YOLOV5S with 640 inputs for primary detection accounted for approximately 30% of the total time, the YOLOV5S with 192 inputs for secondary detection each accounted for approximately 10%, and the ResNet18 model for 11 attribute recognition accounted for approximately 50%. Among these, license plate recognition, vehicle model recognition, and vehicle body attribute recognition accounted for 60% of the frequency.

[0083] 3. Based on the SRAM capacity and model knowledge base, remove or split models that are not suitable for model persistence.

[0084] The YOLOv5S model with 640 inputs for the initial detection is about 28M in size, which exceeds the total capacity of the 16M SRAM on the chip. Therefore, this model is not suitable for optimization and is removed.

[0085] 4. The two secondary detectors with 192 inputs each (each model is approximately 10M in size) of YOLOv5S exceeded half of the 16M SRAM capacity (an empirical value, the actual capacity is between 0.3 and 0.5M). Therefore, based on prior experimental analysis, they were split into two sub-models of 4M and 6M in size. The two 6M sub-models were removed, and the two 4M sub-models (labeled as sub-models 1 and 2) were retained for optimization.

[0086] 5.11 ResNet18 models for attribute recognition (each model is approximately 1M in size) all meet the rule requirements of the model knowledge base and can participate in optimization (labeled as sub-models 3-13 in order).

[0087] 6. Based on steps 3, 4, and 5 above, a set of models participating in the optimization was obtained, consisting of 13 sub-models;

[0088] 7. For each of the 13 sub-models, modify one model parameter at a time according to the configuration in the model optimization parameter library, and then run the business flow to obtain the time consumption, memory usage, power consumption and other parameters of this run; each sub-model has a driver layer parameter during initialization, setting it to True means that the model is persisted, and False means that it is not initialized. There are multiple ways to combine the 13 sub-models. Run the business flow once for each combination to obtain the time consumption, memory usage, power consumption and other parameters of each run, and obtain the solution space of the 13 sub-models.

[0089] 8. Perform a grid search (N^N) on the solution space of the 13 sub-models until the search ends;

[0090] 9. Select the corresponding optimized combination as the final result based on the search results (shortest time).

[0091] The final optimized combination obtained in this embodiment is 1, 2, 3, which means that the two secondary detection sub-models and the vehicle model recognition model are persisted to SRAM as the optimal result. The performance is improved by about 30% compared with the unoptimized version.

[0092] Example 2

[0093] Corresponding to the AI ​​stream multi-model inference optimization method, this application embodiment also provides an inference device, including a processor, an AI chip, and a memory; the memory stores instructions that can be executed by the processor, and the instructions are executed by the processor to enable it to perform the steps of any of the AI ​​multi-model inference optimization methods described above; the AI ​​chip is used to run AI business streams using business streams, and to provide the processor with the percentage and frequency of deep learning model execution time and the performance parameters of the optimized combination required for the inference optimization method.

[0094] Example 3

[0095] This application also provides a readable storage medium, which can be any tangible medium containing or storing a program. The readable storage medium stores a program implementing an AI multi-model inference optimization method. This program is executed by a processor to implement the steps of the AI ​​multi-model inference optimization method as described above. Of course, this program can be used by or in conjunction with an instruction execution system, system, or device. Furthermore, the program code can be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0096] Furthermore, the readable storage medium can be a USB flash drive, or an electrical connection with one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above.

[0097] The aforementioned readable storage medium may be included in the inference device or may exist independently without being assembled into the inference device.

[0098] Example 4

[0099] This application also provides a product, which is a computer program product, including a computer program, which is executed by a processor to implement the steps of the AI ​​multi-model inference optimization method described above.

[0100] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. An AI multi-model inference optimization method, characterized in that, The steps are as follows: S1. Based on the business objectives, determine the AI ​​business flow and obtain the deep learning models used in the AI ​​business flow. The number of deep learning models is Q, where Q≥2. S2, load the unoptimized deep learning models onto the AI ​​chip, and obtain the running time percentage and frequency of each deep learning model after the AI ​​chip runs the AI ​​business flow; the running time percentage and frequency of each deep learning model are obtained by the AI ​​chip's built-in profiling tool when the AI ​​chip uses business data to perform AI business flow inference; S3. Initially eliminate deep learning models with short running time and low frequency to obtain M deep learning models after initial elimination, where M < Q. The initial elimination criteria are: the relationship between the running time t of the deep learning model and the total running time T satisfies t≤cT, 0<c<0.1, and the frequency f of the deep learning model ≤D, D∈[0,3]; S4, select from M deep learning models to obtain a set of models that can be embedded in SRAM, and the set of models contains N optimizable sub-models, where N < M; S5, obtain the solution space of N optimizable sub-models, wherein the solution space is a summary of the performance parameters obtained by running business data on the AI ​​chip for each optimization combination of the N optimizable sub-models; S6. Based on the performance objective, perform a grid search on the solution space. The optimal combination corresponding to the solution that meets the performance objective is the final optimal combination. The corresponding optimizable sub-models in the final optimal combination are stored in SRAM.

2. The AI ​​multi-model inference optimization method according to claim 1, characterized in that: Before determining the AI ​​business flow, a model knowledge base is first established, which stores the processing rules of each deep learning model. The processing rules of each deep learning model are based on prior knowledge to determine whether the deep learning model can be split and the sub-models to be split. The number of deep learning models in the model knowledge base is greater than or equal to the deep learning models used in a certain AI business flow.

3. The AI ​​multi-model inference optimization method according to claim 2, characterized in that: In step S4, the model is filtered based on the relationship between the size of the deep learning model and the total capacity of the SRAM, as well as the model knowledge base. The specific filtering steps are as follows: Compare the relationship between the current deep learning model size 'a' and the total SRAM capacity 'b'. If a > b, then the current deep learning model is not suitable for fixation; If b > a ≥ Kb, K ∈ [0.3, 0.5], then the model is processed according to the processing rules of the current deep learning model in the model knowledge base. If the current deep learning model cannot be split, it is removed. If the current deep learning model can be split, then the two sub-deep learning models are obtained directly, and the sub-deep learning models of different sizes are taken as optimizable sub-models. If a < Kb, K ∈ [0.3, 0.5], then the current deep learning model can be fixed as an optimizable sub-model; After each deep learning model is selected according to the screening method, a set of models that can be embedded in SRAM is obtained. The model set contains N optimizable sub-models, where N < M.

4. The AI ​​multi-model inference optimization method according to any one of claims 1-3, characterized in that: The performance parameters include running time, memory usage, and power consumption.

5. The AI ​​multi-model inference optimization method according to claim 4, characterized in that: The performance target is the minimum value of a certain performance parameter.

6. The AI ​​multi-model inference optimization method according to claim 4, characterized in that: The AI ​​service flows include video structuring service flows, speech recognition service flows, and image optimization service flows.

7. A reasoning device, characterized in that: It includes a processor, an AI chip, and a memory; the memory stores instructions that can be executed by the processor, which, when executed by the processor, enables it to perform the steps of the AI ​​multi-model inference optimization method according to any one of claims 1-6; the AI ​​chip is used to run an AI business flow using a business flow, providing the processor with the required deep learning model's runtime proportion and frequency, and the performance parameters of the optimized combination.

8. A readable storage medium, characterized in that: The readable storage medium stores a program that implements the AI ​​multi-model inference optimization method, and the program that implements the AI ​​multi-model inference optimization method is executed by a processor to implement the steps of the AI ​​multi-model inference optimization method as described in any one of claims 1-6.

9. A product, said product being a computer program product, comprising a computer program, characterized in that: The computer program is executed by a processor to implement the steps of the AI ​​multi-model inference optimization method as described in any one of claims 1-6.

Images