Method and apparatus for elasticity of inference engine fast start and resource recycling

Abstract

The application discloses an elastic method for quick starting and resource recycling of a reasoning engine, and comprises the following steps: obtaining a request log of an edge node in a historical time window, predicting a target model list of a next time window through a prediction model based on the request log; performing a preheating operation on a target model in a non-hot state based on the target model list, so that the state of the target model is at least improved to a warm state; receiving a reasoning request through any one of the target models in the target model list, and making the target model migrate from the warm state to the hot state based on the state of the target model; and driving the target model to degrade from the hot state to the warm state or the cold state through monitoring of resource use data of the target model in the hot state.

Description

Technical Field

[0001] This invention belongs to the field of edge computing and artificial intelligence technology, and in particular relates to a flexible method and apparatus for rapid startup and resource recycling of inference engines. Background Technology

[0002] Currently, edge inference services primarily rely on containerization technology and general inference frameworks. The specific workflow is as follows: when a request arrives, the scheduling system starts a container instance on a node. This instance executes the complete startup chain, including pulling the image, downloading the model file, fully loading the model into GPU memory, and then starting the service. The model lifecycle typically consists of only two coarse-grained states: loading and unloading. Resource scheduling is mostly based on simple real-time metrics or a fixed number of replicas. While the above approach is feasible in stable cloud environments, its application in distributed edge scenarios exposes the following inherent drawbacks: First, the complete startup process results in a delay of minutes or even hours for the service to go from zero to availability, which cannot meet the second-level response requirements for sudden traffic in edge scenarios.

[0003] Second, it only has a load / unload binary state and lacks the ability to manage and utilize the intermediate states of the model in multiple storage media such as disk, memory, and video memory, resulting in inflexible resource occupation and release.

[0004] Third, scheduling decisions rely entirely on real-time requests, which is a passive response mode. It cannot predict future loads based on historical patterns, thus failing to prepare the model in advance to an intermediate state that is about to be usable, causing the system to always start responding to peak loads from the slowest, coldest state.

[0005] Fourth, in order to ensure service quality, the long-tail model with low-frequency access must also reside in video memory, which squeezes out the valuable video memory resources that could be used to process high-frequency requests, resulting in significant resource waste.

[0006] In summary, existing technical solutions suffer from systemic deficiencies in three dimensions: response agility, resource utilization efficiency, and intelligent scheduling. The root cause lies in directly applying a cloud-based, stable, and coarse-grained service deployment model to edge scenarios. Therefore, there is an urgent need for an innovative approach that can decouple and preheat the inference engine startup process, manage the model state with fine-grained and forward-looking capabilities, and dynamically reclaim computing power to systematically address the unique challenges of edge inference. Summary of the Invention

[0007] To address the aforementioned issues, this invention proposes a flexible method and apparatus for rapid startup and resource reclamation of inference engines. Through a multi-level preheating mechanism, cold start time is compressed from minutes to seconds, achieving a significant reduction in startup time. Based on memory swapping technology and predictive expansion strategies, the response speed to sudden traffic surges is improved, enabling rapid service expansion. Dynamic switching between hot and warm model states on demand effectively avoids memory waste for long-tail models, improving GPU utilization. Through a layered storage architecture of warm and cold states, this invention enhances the node's capacity to accommodate massive amounts of long-tail, low-frequency models, maximizing resource scheduling. Predictive resource reservation and rapid response mechanisms effectively mitigate the risk of service avalanche, ensuring end-to-end response stability and SLA compliance in high-concurrency scenarios.

[0008] A first aspect of the present invention provides a flexible method for fast startup and resource reclamation of an inference engine, comprising: Obtain the request logs of edge nodes within the historical time window, and predict the target model list for the next time window based on the request logs using a prediction model. Based on the list of target models, a preheating operation is performed on the target models that are in a non-thermal state to raise their state to at least a warm state. Receive inference requests by any one of the target models in the target model list, and based on the state of the target model, migrate the target model from a warm state to a hot state; By monitoring the resource usage data of the target model in a hot state, the target model is driven to degrade from a hot state to a warm or cold state.

[0009] Preferably, it further includes: deploying an inference service model for each edge computing node, wherein the inference service model has a multi-level cache state machine, and the state machine includes at least an empty state, a cold state, a warm state, and a hot state; wherein, the cold state is used for storing the resources of the inference service model on the disk, the warm state is used for loading the inference service model into memory and completing the inference session initialization, and the hot state is used for loading the GPU memory of the inference service model.

[0010] Preferably, the step of obtaining the request logs of the edge nodes within the historical time window, and predicting the target model list for the next time window based on the request logs using a prediction model, further includes: Get the request logs of the edge nodes for the past seven days; Features are extracted from the request logs using a hybrid prediction model combining LSTM and Poisson distribution. The actual number of instances is obtained based on the aforementioned features. The calculation logic includes: dividing the total number of requests per hour by the concurrency of each instance to obtain a base value, and then adding a configured error term based on the base value. Statistical analysis of P50, P90, and P99 delay indices at each data acquisition time; Based on the aforementioned P50, P90, and P99 latency metrics, and combined with the number of instances of the inference service model at each stage, a Top-K model list is obtained through scaling up and down analysis.

[0011] Preferably, the features include hourly request volume categorized according to the model, business periodicity characteristics, and sudden peak characteristics.

[0012] Preferably, the preheating operation includes at least one of container image pulling, model weight downloading, file system pre-reading, GPU environment dependency checking, establishing memory mapping for model weights, and generating inference engine compilation cache.

[0013] Preferably, establishing a memory mapping for model weights further includes: performing the steps of decompressing the prediction model weight parameters and loading them into system memory, initializing the inference engine session, and initializing the KV-Cache infrastructure.

[0014] Preferably, the step of migrating from a warm state to a hot state based on the state of the target model further includes: Monitor the actual inference request load. If the request volume exceeds the current peak processing capacity, copy the weight data and execution environment parameters of the target model that is in a warm state from memory to GPU memory. If the GPU memory copying is completed, control the target model to be put into inference service.

[0015] Preferably, the step of driving the target model to degrade from a hot state to a warm or cold state by monitoring resource usage data of the target model in a hot state further includes: Monitor the idle time period of the target model in the hot state. If no inference request is received within the preset first time threshold, release the GPU video memory resources occupied by the target model and retain its data in memory, so that the model state is downgraded from hot state to warm state. If no inference request is received within the preset second time threshold period during the idle time period of the target model in the warm state, the weight data of the target model is written back from memory to disk storage, so that the target model state is downgraded from the warm state to the cold state.

[0016] A second aspect of the present invention provides a flexible device for rapid startup and resource recovery of an inference engine, comprising: The prediction module is used to obtain the request logs of edge nodes within the historical time window, and predict the target model list for the next time window based on the request logs using the prediction model. The preheating module is used to perform a preheating operation on the target models that are in a non-thermal state based on the target model list, so that their state is raised to at least a warm state; The state transition module is used to receive inference requests through any one of the target models in the target model list, and to transition from a warm state to a hot state through the target model based on the state of the target model. The resource recycling module is used to drive the target model to degrade from a hot state to a warm or cold state by monitoring the resource usage data of the target model in a hot state.

[0017] A third aspect of the present invention provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the flexible method for fast startup and resource reclamation of the inference engine as described in any of the preceding claims.

[0018] Because the present invention adopts the above technical solution, it has the following advantages and positive effects compared with the prior art: Through a multi-level preheating mechanism, cold start time is reduced from minutes to seconds, achieving a significant reduction in startup time. Based on memory swapping technology and predictive expansion strategies, the response speed to sudden traffic surges is improved, enabling ultra-fast service expansion.

[0019] By dynamically switching between hot and cool states of the model as needed, the waste of GPU memory in long-tailed models is effectively avoided, and the GPU utilization rate is improved.

[0020] By employing a layered storage architecture with warm and cold states, this invention enhances the node's capacity to accommodate massive amounts of long-tailed, low-frequency models, thereby maximizing resource scheduling.

[0021] Through predictive resource reservation and rapid response mechanisms, the risk of service avalanche is effectively avoided, ensuring end-to-end response stability and SLA compliance in high-concurrency scenarios. Attached Figure Description

[0022] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings, wherein: Figure 1 This is the main flowchart of the flexible method for fast startup and resource recycling of the inference engine of the present invention; Figure 2 This is an embodiment of the flexible method for fast startup and resource recycling of the inference engine of the present invention. Detailed Implementation

[0023] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. The advantages and features of the present invention will become clearer from the following description and claims. It should be noted that the drawings are all in a very simplified form and use non-precise ratios, and are only used to facilitate and clarify the illustration of the embodiments of the present invention.

[0024] It should be noted that all directional indicators (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative positional relationship and movement of each component in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indicator will also change accordingly.

[0025] First Embodiment See Figure 1 and Figure 2 The first aspect of the present invention provides a flexible method for fast startup and resource reclamation of an inference engine, comprising: Obtain the request logs of edge nodes within the historical time window, and predict the target model list for the next time window based on the request logs using a prediction model. Based on the list of target models, perform a preheating operation on the target models that are in a non-thermal state to raise their state to at least a warm state. Receive inference requests from any target model in the target model list, and based on the state of the target model, migrate the target model from a warm state to a hot state; By monitoring the resource usage data of the target model in a hot state, the target model is driven to degrade from a hot state to a warm or cold state.

[0026] This invention achieves systematic optimization through a closed-loop process of prediction-driven, state transition, and dynamic recycling. The specific process is as follows: In the prediction and preheating stage, see... Figure 2 The specific process involves three states: empty, cold, and warm. In the cold state, resources for the inference service model are stored on disk. In the warm state, the inference service model is loaded into memory and the inference session is initialized. In the hot state, the inference service model is loaded into GPU (Graphics Processing Unit) memory. This reduces the service cold start time to the second level. By using a predictive model based on historical logs to pre-identify the target model and asynchronously performing preheating operations such as image retrieval, model download, and memory loading in the background, the model is already in a warm state before the request arrives. This eliminates the long wait from the empty state in traditional solutions. Starting from the warm state during the request triggering and rapid swapping phase enables scalability for sudden traffic surges. When the actual inference request arrives, since the target model is already pre-positioned in the warm state, the system only needs to perform the operation of quickly migrating the model from memory to GPU memory. This process is short, allowing service processing capacity to increase with traffic spikes and improving response speed. During the dynamic monitoring and resource swapping phase, the system achieves hot, warm, and cold state transitions, enabling efficient utilization and dynamic recycling of GPU memory. This significantly improves resource utilization, ensures that valuable GPU memory always prioritizes active, high-frequency requests, avoids invalid occupation by long-tail models, and enhances GPU utilization.

[0027] Preferably, it also includes: deploying an inference service model for each edge computing node, wherein the inference service model has a multi-level cache state machine, and the state machine includes at least an empty state, a cold state, a warm state, and a hot state; wherein, the cold state is used for storing the resources of the inference service model on the disk, the warm state is used for loading the inference service model into memory and completing the inference session initialization, and the hot state is used for loading the inference service model into GPU memory.

[0028] By defining temperature states, the most time-consuming initialization work can be completed without consuming GPU memory, achieving the two stages of service preparation and peak computing power readiness. Cold state has the lowest cost but the slowest activation; temperature state has a moderate cost but can be activated quickly; hot state has the highest cost but the fastest response. This quantitative definition provides a clear decision-making basis and objective for system scheduling strategies, moving away from vague judgments based on experience. By explicitly defining the differences between temperature and hot states, second-level memory swapping can be performed to handle sudden traffic surges; by defining cold states, long-idle models can be written back from memory to disk, safely releasing memory.

[0029] Preferably, the step of obtaining the request logs of the edge nodes within the historical time window, and predicting the target model list for the next time window based on the request logs using a prediction model, further includes: Get the request logs of the edge nodes for the past seven days; Features are extracted from request logs using a hybrid prediction model combining LSTM and Poisson distribution. The actual number of instances is obtained based on features. The calculation logic includes: dividing the total number of requests per hour by the concurrency of each instance to obtain a base value, and adding the configured error term based on the base value. The P50 (50th cold percentile, median), P90 (90th cold percentile, 90th percentile), and P99 (99th cold percentile, 99th percentile) delay indices were statistically analyzed at each data collection time. Based on the P50, P90, and P99 latency metrics, and combined with the number of instances of the inference service model at each stage, a Top-K (Top-K cold items, the top K items) model list is obtained through scaling up and down analysis.

[0030] By employing a hybrid prediction model combining LSTM and Poisson distribution, the limitations of traditional single prediction algorithms in scenarios with weak edge computing power, multiple models, and large fluctuations are overcome, improving prediction accuracy and resource preparation hit rate. The solution innovatively combines the basic computing power requirements of total request volume / instance concurrency capacity with P50 / P90 / P99 service latency metrics and the number of instances in each stage of the model for scaling analysis, achieving optimal warm-up costs. Through prediction and capacity calculation, combined with dynamic adjustments based on latency metrics, unnecessary resource consumption is reduced from the source, improving overall GPU utilization.

[0031] Preferably, the features include hourly request volume categorized by the model, business periodicity, and sudden peak characteristics.

[0032] By extracting hourly request volumes segmented by model, predictions can learn independent access patterns for each model, solving the problem of unpredictable traffic in multi-model scenarios. Incorporating business periodic features allows the model to understand and learn regular fluctuations caused by human activity or business operations. Special attention is paid to sudden peak features, enabling the algorithm to identify and predict, to some extent, non-periodic, sudden traffic surges. This combination of features characterizes the core dimensions of edge inference traffic, enhancing predictive capabilities. Based on this fine-grained and multi-dimensional feature set, future traffic is predicted using a hybrid LSTM and Poisson model. The reliability of the output Top-K model list is significantly enhanced. Warming up the Top-K model list from an empty state to a cold state and loading it into memory from a cold state to a warm state allows valuable warming resources to be more concentrated on models with a high probability of being accessed, reducing wasted resources and mitigating the risk of cold starts when needed but not warmed up.

[0033] Preferably, the preheating operation includes at least one of the following: container image pulling, model weight downloading, file system pre-reading, GPU environment dependency checking, establishing memory mapping for model weights, and generating inference engine compilation cache.

[0034] This preferred solution eliminates network transmission and cold I / O latency bottlenecks when inference requests arrive, ensuring that data is already in memory or file system cache for rapid retrieval. Simultaneously, by pre-establishing memory mapping and generating an inference engine compilation cache, it avoids the time-consuming just-in-time compilation process and memory mapping overhead when the deep learning framework first loads the model. Furthermore, pre-checking cold GPU environment dependencies eliminates the risk of runtime environment configuration errors. These pre-warming measures forward the initialization computation and cold I / O load, ensuring that the model does not need to wait for resource loading when receiving subsequent requests, significantly reducing initialization latency.

[0035] Preferably, establishing a memory mapping for model weights further includes: performing the steps of decompressing the prediction model weight parameters and loading them into system memory, initializing the inference engine session, and initializing the KV-Cache infrastructure.

[0036] By pre-compiling and loading model weights, as well as initializing the inference engine session and KV-Cache structure during the warm-up phase, CPU computation and disk I / O operations are completely decoupled from the real-time processing path of inference requests. Model weights are pre-loaded into system memory, eliminating disk read latency upon request arrival and runtime decompression overhead. Pre-initializing the inference engine session and KV-Cache infrastructure avoids the time-consuming build and memory planning process during the first run of the deep learning framework. The process of migrating the model from a warm state to a hot state is simplified to high-speed memory-to-GPU memory data copying, achieving rapid inference service startup without continuously consuming GPU memory resources, thus balancing system response speed and resource costs.

[0037] Preferably, the step of transferring the state from a warm state to a hot state based on the target model further includes: Monitor the actual inference request load. If the request volume exceeds the current peak processing capacity, copy the weight data and execution environment parameters of the target model that is in a warm state from memory to GPU memory. If the GPU memory copying is completed, control the target model to be put into inference service.

[0038] When the monitoring system detects that the real-time request load exceeds the preset capacity, it initiates a rapid, deterministic state transition. Since the target model is already in a warm state, the system only needs to perform a high-speed DMA data copy from host memory to GPU memory. This process is short, allowing service computing power to quickly match the surge in load, fundamentally eliminating the risks of request queuing, response time spikes, and even service avalanches caused by resource expansion delays. It provides reliable technical support for handling sudden traffic surges and ensures that the Service Level Agreement (SLA) remains stable under pressure. This step, predictive driver warm-up and multi-level state caching, is the final and most critical output link in the entire intelligent scheduling chain, ensuring that the system's intelligence is translated into performance without loss. This allows the design goal of the entire four-level state machine architecture to perfectly achieve the optimal balance between agility and economy.

[0039] Preferably, the step of driving the target model to degrade from a hot state to a warm or cold state by monitoring resource usage data of the target model in a hot state further includes: Monitor the idle time period of the target model in the hot state. If no inference request is received within the preset first time threshold, release the GPU video memory resources occupied by the target model and retain its data in memory, so that the model state is downgraded from hot state to warm state. Monitor the idle time period of the target model in the warm state. If no inference request is received within the preset second time threshold, write the weight data of the target model back from memory to disk storage, so that the target model state is downgraded from the warm state to the cold state.

[0040] By setting a first duration threshold, the system can quickly detect the idle state of a model after it stops serving and automatically trigger hot and warm state degradation operations to immediately release the expensive GPU memory it occupies. This transforms the GPU memory from being exclusively occupied by a few models for a long time to a high-speed shared resource pool that can be contested by all models on demand. This enables efficient recycling of GPU memory resources among multiple models, avoiding the ineffective locking of computing power by low-frequency long-tail models, thereby improving the overall GPU utilization to a high level (e.g., >80%). This is a key solution to the core pain point in the background technology.

[0041] Second Embodiment A second aspect of the present invention provides a flexible device for rapid startup and resource recovery of an inference engine, comprising: The prediction module is used to obtain the request logs of edge nodes within the historical time window, and predict the target model list for the next time window based on the request logs using the prediction model. The preheating module is used to perform preheating operations on target models that are in a non-thermal state based on the list of target models, so that their state is at least raised to a warm state. The state transition module is used to receive inference requests from any target model in the target model list and to transition the state from a warm state to a hot state based on the target model. The resource recycling module is used to drive the target model to degrade from a hot state to a warm or cold state by monitoring the resource usage data of the target model in a hot state.

[0042] This device, through the coordinated operation of four core modules—prediction, preheating, migration, and recycling—solidifies the dynamic scheduling method into a deployable and highly available system. It achieves closed-loop automation from prediction-driven decision-making to resource scaling, ensuring real-time service response and high stability while systematically and consistently improving the utilization rate of computing resources such as GPUs.

[0043] Third Embodiment A third aspect of the present invention provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the flexible method for rapid startup and resource reclamation of the inference engine described above.

[0044] This electronic device, through hardware and software collaboration, solidifies and implements the aforementioned dynamic scheduling method into a standardized, independently operable computing unit. It provides the physical carrier and operating environment for the entire invention, enabling the method to be executed stably and efficiently on actual edge servers, computing nodes, and other hardware. This combines theoretical algorithms with engineering practice, ultimately resulting in real-time startup, improved resource utilization, and enhanced verifiable practical performance.

[0045] In the description of this application, it should be noted that the terms "inner" and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship commonly used when the product is in use. They are used only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation on this application. In addition, the terms "first," "second," etc., are only used to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0046] It should also be noted that, unless otherwise explicitly specified and limited, the terms "setup" and "connection" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0047] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific identification content executed by the system and device described above can be referred to the corresponding process in the foregoing method embodiments.

[0048] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings, but the present invention is not limited to the above embodiments. Even if various changes are made to the present invention, if these changes fall within the scope of the claims of the present invention and their equivalents, they shall still fall within the protection scope of the present invention.

Claims

1. A flexible method for fast startup and resource reclamation of an inference engine, characterized in that, include: Obtain the request logs of edge nodes within the historical time window, and predict the target model list for the next time window based on the request logs using a prediction model. Based on the list of target models, a preheating operation is performed on the target models that are in a non-thermal state to raise their state to at least a warm state. Receive inference requests by any one of the target models in the target model list, and based on the state of the target model, migrate the target model from a warm state to a hot state; By monitoring the resource usage data of the target model in a hot state, the target model is driven to degrade from a hot state to a warm or cold state.

2. The flexible method for rapid startup and resource reclamation of the inference engine according to claim 1, characterized in that, Also includes: An inference service model is deployed for each edge computing node. The inference service model has a multi-level cache state machine, which includes at least an empty state, a cold state, a warm state, and a hot state. The cold state is used for storing the resources of the inference service model on the disk. The warm state is used for loading the inference service model into memory and completing the inference session initialization. The hot state is used for loading the inference service model into GPU memory.

3. The flexible method for rapid startup and resource reclamation of the inference engine according to claim 1, characterized in that, The step of obtaining the request logs of edge nodes within a historical time window and predicting the target model list for the next time window based on the request logs using a prediction model further includes: Get the request logs of the edge nodes for the past seven days; Features are extracted from the request logs using a hybrid prediction model combining LSTM and Poisson distribution. The actual number of instances is obtained based on the aforementioned features. The calculation logic includes: dividing the total number of requests per hour by the concurrency of each instance to obtain a base value, and then adding a configured error term based on the base value. Statistical analysis of P50, P90, and P99 delay indices at each data acquisition time; Based on the aforementioned P50, P90, and P99 latency metrics, and combined with the number of instances of the inference service model at each stage, a Top-K model list is obtained through scaling up and down analysis.

4. The flexible method for rapid startup and resource reclamation of the inference engine according to claim 3, characterized in that, The features include hourly request volume categorized by model, business periodicity, and sudden peak characteristics.

5. The flexible method for rapid startup and resource reclamation of the inference engine according to claim 1, characterized in that, The preheating operation includes at least one of the following: container image pulling, model weight downloading, file system pre-reading, GPU environment dependency checking, establishing memory mapping for model weights, and generating inference engine compilation cache.

6. The flexible method for rapid startup and resource reclamation of the inference engine according to claim 5, characterized in that, Establishing a memory mapping for model weights further includes performing the steps of decompressing and loading the predicted model weight parameters into system memory, initializing the inference engine session, and initializing the KV-Cache infrastructure.

7. The flexible method for rapid startup and resource reclamation of the inference engine according to claim 1, characterized in that, The step of migrating from a warm state to a hot state based on the state of the target model further includes: Monitor the actual inference request load. If the request volume exceeds the current peak processing capacity, copy the weight data and execution environment parameters of the target model that is in a warm state from memory to GPU memory. If the GPU memory copying is completed, control the target model to be put into inference service.

8. The flexible method for rapid startup and resource reclamation of the inference engine according to claim 1, characterized in that, The step of driving the target model to degrade from a hot state to a warm or cold state by monitoring resource usage data of the target model in a hot state further includes: Monitor the idle time period of the target model in the hot state. If no inference request is received within the preset first time threshold, release the GPU video memory resources occupied by the target model and retain its data in memory, so that the model state is downgraded from hot state to warm state. If no inference request is received within the preset second time threshold period during the idle time period of the target model in the warm state, the weight data of the target model is written back from memory to disk storage, so that the target model state is downgraded from the warm state to the cold state.

9. A flexible device for rapid startup and resource recycling of an inference engine, characterized in that, include: The prediction module is used to obtain the request logs of edge nodes within the historical time window, and predict the target model list for the next time window based on the request logs using the prediction model. The preheating module is used to perform a preheating operation on the target models that are in a non-thermal state based on the target model list, so that their state is raised to at least a warm state; The state transition module is used to receive inference requests through any one of the target models in the target model list, and to transition from a warm state to a hot state through the target model based on the state of the target model. The resource recycling module is used to drive the target model to degrade from a hot state to a warm or cold state by monitoring the resource usage data of the target model in a hot state.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the computer program is executed by the processor, it implements the steps of the flexible method for fast startup and resource reclamation of the inference engine as described in any one of claims 1-8.

Images