Container video memory budget control and dynamic optimization system and method for ASIC edge equipment

Abstract

The invention provides a container video memory budget control and dynamic optimization system and method for ASIC edge equipment, and the system comprises a video memory statistics and quota interface, a container identification mapping layer, a monitoring and decision engine, a strategy engine, an actuator, and a monitoring and auditing module. The method comprises the following steps: establishing a video memory budget contract by taking a container as an object and finishing container identifier mapping; a container-oriented video memory statistics and quota interface is provided in a driving layer or a runtime library layer; operating a resident video memory daemon process cycle in a user mode to collect video memory occupation and pressure indexes of each container, and generating an optimization action in combination with a service priority and a threshold rule; when a video memory pressure event is triggered, a combined action of video memory budget shaping, reasoning parameter adaptive adjustment and cache recovery is performed on a target container through an actuator, and the effect is rechecked and audited by adopting a monitoring and auditing module, so that the stability and continuity of multi-container concurrent reasoning in an edge device resource boundary are improved.

Description

Technical Field

[0001] This invention relates to the field of container memory optimization technology, specifically to a container memory budget control and dynamic optimization system and method for ASIC edge devices. Background Technology

[0002] With the increasing prevalence of edge intelligence inference services, multiple AI applications, such as image recognition, object detection, structured data extraction, and speech analysis, often run concurrently on a single edge device equipped with an ASIC inference acceleration chip. These applications are typically deployed in containers to achieve isolation, rapid delivery, and ease of operation and maintenance.

[0003] Unlike data center servers, edge devices typically have stronger resource limitations: accelerator-side available GPU / device memory capacity is limited, bandwidth and power consumption are constrained, and swap space is lacking or swap overhead is unacceptable. During concurrent inference, model weights, activation cache, KV cache optimization, and operator temporary buffers can create peak memory usage in ASIC devices. Once a device experiences an Out Of Memory (OOM) error, it can lead to inference failure, container restarts, or even driver anomalies or device resets, causing other containers on the same machine to also be interrupted, creating a single point of OOM that impacts the entire machine's operational risks.

[0004] For general-purpose architectures such as x86 / ARM, container resource isolation mainly relies on cgroups (control groups) to limit CPU / memory, which can achieve a relatively stable isolation effect. For GPU scenarios, the mature memory management ecosystem (such as memory pools, streaming allocation, job-level memory limits, and driver-side statistics interfaces) can be used in engineering to achieve a certain degree of observability and limitation. However, on ASIC edge devices, driver interfaces and runtime ABIs often exhibit vendor differences, and memory allocation is often hidden inside the inference runtime. The container layer has difficulty knowing who is requesting memory, how much is requested, and whether it is reclaimable. This makes it impossible for existing container mechanisms to form container-level hard boundary control over accelerator memory, resulting in the inability to guarantee the stability of multi-container concurrent inference. Summary of the Invention

[0005] To address the problems of uncontrollable video memory / device memory, frequent OOM (Out of Memory) errors, and difficulty in ensuring service continuity under multi-container concurrent inference in existing ASIC edge devices, this invention provides a system and method for container video memory budget control and dynamic optimization for ASIC edge devices.

[0006] This invention relates to a container memory budget control and dynamic optimization system for ASIC edge devices, comprising: Memory Statistics and Limit Interface: Set at the input of the ASIC edge device scheduler, used for container registration, and memory control is performed on the container as the object; Container Identifier Mapping Layer: Used to map runtime information provided by the container runtime to container handles that can be recognized by the driver / runtime library, enabling memory statistics and budget control on a container-by-container basis; Monitoring and Decision Engine: Used to periodically collect memory usage-related indicators for each container, calculate the memory pressure index based on the memory usage-related indicators, and trigger the strategy engine action if the memory pressure index exceeds the memory pressure threshold. Strategy Engine: Used to define memory pressure thresholds and optimize action execution rules, and output executable optimized action sequences to the executor; Executor: Located between the strategy engine and the memory statistics and limit interface, it is used to perform budget shaping, trigger inference runtime cache reclamation, and adaptively adjust inference parameters according to the optimized action sequence; The monitoring and auditing module is connected to the output of the detection and decision engine and the memory statistics and limit interface, respectively. It is used to collect memory indicators, optimize actions and effect verification results, and issue alarms for memory pressure events and risks caused by insufficient memory.

[0007] Furthermore, it also includes a performance graph module: used to perform correlation analysis between memory usage-related indicators, memory pressure index and optimization action sequences, providing calibration data for the strategy engine.

[0008] Furthermore, it also includes a memory usage model building module: used to generate a memory usage model based on container runtime data, and output peak budget suggestions and phased budget suggestions.

[0009] Furthermore, it also includes a budget execution interception layer: deployed at the driver interface or user-space memory allocator location, used to perform budget verification on the container's memory requests, and to intercept requests when the requests cause memory to exceed the budget.

[0010] Furthermore, after the budget execution interception layer intercepts, it returns an error code with the container handle and request context, and the monitoring and decision engine triggers the corresponding degradation action; when the budget is allowed but the global constraints will be violated, the monitoring and decision engine shrinks the budget of low-priority containers according to priority, and then allows high-priority containers to continue to allocate.

[0011] This invention also provides a method for container memory budget control and dynamic optimization, based on the aforementioned container memory budget control and dynamic optimization system for ASIC edge devices, comprising the following steps: S1: After the container starts, the monitoring and decision engine obtains relevant indicators of each container's video memory usage, obtains the container handle, and establishes a mapping during container runtime; S2: The policy engine calculates the initial budget for each container based on the total device memory, security reserve, and container priority, and then sends it to the executor; S3: The executor issues the initial budget of each container to the memory statistics and limit interface, and then performs memory scheduling through the ASIC edge device scheduler and hardware resources; S4: The monitoring and decision engine collects the memory usage, allocation failure count and critical latency of each container at a preset period, calculates the memory pressure index, and forms statistics within a time window. When the memory pressure index exceeds the memory pressure threshold or an allocation failure occurs, the strategy engine is triggered. S5: The strategy engine outputs an executable sequence of optimized actions to the executor, the sequence of optimized actions including cache reclamation and inference parameter adjustment items; S6: The executor performs budget shaping according to the action sequence, triggers inference runtime cache reclamation, adaptively adjusts inference parameters, and calculates performance scores. S7: The monitoring and auditing module reads the optimized memory usage indicators and the new video memory pressure index in the review window. If the pressure is relieved, it returns to step S4. If the pressure is not improved or affects the service level agreement (SLA), it rolls back to the previous policy or performs a downgrade and issues an alarm.

[0012] Furthermore, it also includes graph generation and strategy calibration steps: periodically aggregating the metrics and action records of each container, generating a performance graph of container-level time series and associated views through the performance graph module, and calibrating the various threshold and weight parameters of the design based on the samples in the performance graph.

[0013] Furthermore, the method by which the strategy engine calibrates the various threshold and weight parameters of the design based on samples in the performance graph is as follows: (1) Set the sample points S={M} of each container in the performance graph. i B i F i L i Action, E i P i} is the input, where i is the index of the container, and M is the index of the container. i B represents the video memory usage of the i-th container. i For the memory budget of the i-th container, F i L is the allocation failure count for the i-th container. i Let E be the critical latency of the i-th container, and Action be the optimization action. i P is the score for the optimization effect of the i-th container. i Let be the memory pressure index of the i-th container; (2) Observe the memory usage trend by sliding average within a fixed time window. If a certain type of action generally brings improvement, then appropriately relax the trigger conditions or increase the priority of the action; if it is ineffective or causes jitter, then tighten the trigger conditions or reduce the priority of the action, and bind all adjustments with calibration, action recording and calibration effects.

[0014] Furthermore, in step S2, the initial budget is calculated as follows: based on the total device memory... Deducting safety margin Then, according to the priority of container i and historical baseline The available budget is allocated proportionally, and the result is truncated at... The interval, where, , Let $i$ be the minimum and maximum budget allowed for container $i$, respectively.

[0015] Furthermore, during the operational phase, the target budget Without violating the overall constraints, the system makes small, automatic adjustments based on the difference between the current memory usage and the target budget, allocation failure signals, and memory stress levels, ensuring the result is always limited to... Define an optimization performance score for panel presentation and strategy calibration. The strategy engine introduces a hysteresis window and a review window in the determination of memory stress index, and restricts budget updates to... Within the interval; when global constraint reclamation is triggered, lower-priority containers are stratified and contracted according to priority. The global constraint formula is:

[0016] in, For the number of concurrent containers, This is a safety reserve for drivers, systems, and emergency buffers.

[0017] Compared with existing technologies, the beneficial effects of this invention are as follows: This invention can form a container-level hard boundary for video memory, enabling each container to have a configurable upper limit for video memory budget and reducing mutual interference; through video memory pressure monitoring and dynamic optimization executors, the model inference parameters and caching strategies can be controlled and adjusted during runtime, avoiding frequent restarts and device resets, thereby alleviating video memory pressure on ASIC edge devices without interrupting containers; in addition, this invention dynamically shapes the video memory budget based on business priority and risk thresholds, ensuring that critical tasks are guaranteed when video memory is scarce, while limiting the peak usage of low-priority tasks; video memory usage is attributed to container objects and audit records are formed, supporting the traceability of OOM risks and optimization actions, facilitating the operation and maintenance reproduction and strategy iteration of batch devices. Attached Figure Description

[0018] To more clearly illustrate the solutions in this invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 This is a system framework diagram of the present invention; Figure 2 This is a flowchart of the container memory budget control and dynamic optimization method of the present invention. Detailed Implementation

[0020] Unless otherwise defined, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains; the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings are used to distinguish different objects, not to describe a particular order.

[0021] In this invention, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment to other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this invention can be combined with other embodiments.

[0022] To enable those skilled in the art to better understand the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0023] like Figure 1As shown, the container memory budget control and dynamic optimization system for ASIC edge devices of this invention includes a memory statistics and quota interface, a container identifier mapping layer, a monitoring and decision engine, a policy engine, an executor, and a monitoring and auditing module. In this example, the monitoring and decision engine is implemented by a memory daemon. This invention establishes a memory budget contract for containers and completes container identifier mapping; it provides a container-oriented memory statistics and quota interface at the driver layer or runtime layer; it runs a resident memory daemon in user space to periodically collect memory usage and pressure indicators of each container, and generates optimization actions by combining business priorities and threshold rules; when a memory pressure event is triggered, the executor performs a combination of actions on the target container, including memory budget shaping, adaptive adjustment of inference parameters, and cache reclamation, and the monitoring and auditing module verifies and audits the effects, thereby improving the stability and continuity of multi-container concurrent inference within the resource boundaries of edge devices.

[0024] Preferably, the present invention further includes a memory usage model construction module: used to generate a memory usage model based on container runtime data, and output peak budget suggestions and phased budget suggestions. Before budget allocation, the memory daemon process of the present invention constructs a memory usage model for tasks in different containers, and uses the model output as the basis for budget allocation and dynamic shaping; simultaneously, it intercepts over-budget requests on the driver side or memory manager side and returns attributable error contexts, making the budget not only a monitoring metric but also an executable hard constraint.

[0025] Preferably, the present invention further includes a performance graph module to construct a performance graph, provide calibration data for the policy engine and an interpretable chain of evidence for operation and maintenance, and a budget execution interception layer for budget verification of the container's memory requests, and to perform corresponding interception actions when the requests result in exceeding the budget.

[0026] In this invention, ASIC video memory / device memory refers to the storage space available for inference execution on the ASIC accelerator side. It can be on-chip SRAM, off-chip DDR, HBM, etc. In this invention, it is used to hold model weights, activation cache, KV cache, and operator temporary buffers. The video memory daemon in this example is a resident background process that plays a crucial role in the closed-loop control of monitoring, decision-making, execution, and verification in this invention.

[0027] The functions of each module in this invention are as follows: 1. Memory Statistics and Quota Interface: Used for memory control based on containers, supporting at least container registration, memory usage reading, memory budget writing / updating, and allocation failure event querying. The interface can be implemented using IOCTL, sysfs, or character devices, etc.

[0028] 2. Container Identifier Mapping Layer: This layer maps information such as container_id (container ID), cgroup path, and main process PID provided by the container runtime to container_handle (container handle) that can be recognized by the driver / runtime library, so as to realize memory statistics and budget control on a container-by-container basis.

[0029] 3. Monitoring and Decision Engine (Memory Daemon): Used to periodically collect memory usage, allocation failure counts and key latency indicators of each container, calculate memory pressure index and trigger policy evaluation, generate optimization actions and schedule executors to take effect, and at the same time record audit logs and output monitoring data.

[0030] 4. Strategy Engine: Used to define rules such as budget upper / lower limits, business priorities, memory pressure thresholds, hysteresis windows and safety reserves, and output executable optimization action sequences.

[0031] 5. Optimize the executor: This is used to encapsulate the calling logic of the video memory interface and inference runtime. It should at least support budget shaping, video memory reclamation (cleaning up the cache / releasing the free memory pool), and inference parameter adjustment (such as dynamically adjusting the batch size, KV cache limit, precision mode or pre-allocation strategy), and provide idempotent delivery, failure retry and rollback / degradation strategies.

[0032] 6. Monitoring and Auditing Module: Used to collect memory metrics, review optimization actions and effects, and issue alerts for memory pressure events and OOM risks.

[0033] 7. Performance Graph Module: Used to display container-level video memory usage in M... i Video memory budget B i Allocation failure count F i Critical latency L i Memory stress index P i The sample sequence composed of optimization actions is visualized and correlated with a unified system, and the output includes performance graphs of containers, time, metrics and actions. This provides calibration data for the policy engine and an interpretable chain of evidence for operations and maintenance.

[0034] 8. Memory Model Building Module: Used for monitoring and analysis based on container runtime data to identify performance bottlenecks and optimization points in the program, thereby improving the program's running efficiency and performance. It also generates a memory usage model based on online probing or rule-based upper bound estimation, and outputs peak budget suggestions and phased budget suggestions. When the model confidence is insufficient, it outputs a conservative upper bound to avoid OOM risk.

[0035] 9. Budget Execution Interception Layer: Deployed at the driver interface or user-space memory allocator, it is used to perform budget verification on the memory requests of containers; when the request results in exceeding the budget, it executes one or a combination of rejection / blocking / degradation, and writes the error code and context into the audit chain to support attribution and recovery.

[0036] Preferably, in order to enable the performance graph of the present invention to have an implementable data structure and a reproducible attribution metric, the present invention defines the performance graph as a graph data object perf_graph composed of three elements: indicator time series, event / action chain, and attribution correlation, which includes at least: (1) Graph entity set, including container entities, model entities, stage entities, action entities, event entities, etc.; (2) Graph relationship set, including the model objects of container operation, inference phase (loading / warming / inference / unloading), stress event-triggered optimization actions, action objects, action failures or introduced abnormal events and container indicator time sequence attribution, etc. (3) Unify the recording standards for events and actions: The video memory daemon generates an event_id for each trigger and an action_id for each execution. It also forces the container_handle, timestamp, budget_revision, action type and parameter snapshot, return code, and review window statistics to be written into the audit log, so that any pressure increase - action execution - effect change can be associated and reproduced by the performance graph. (4) Indicator sampling and aggregation rules: In order to adapt to the resource boundaries of ASIC edge devices, this example collects the container-level video memory usage M at a period of T during container runtime. i Video memory budget B i Memory stress index P i Basic indicators; when the memory pressure index P is detected i When the maximum limit is exceeded or an allocation failure occurs, the sampling is encrypted at a period of T within the review window and the difference before and after the action is recorded, thereby improving the attribution accuracy of critical events without significantly increasing normal overhead.

[0037] To achieve flexible adjustment of the performance graph, this invention configures the output scope and visualization dimensions of the performance graph: the strategy engine maintains a graph configuration (graph_profile) for each type of container or model, which includes at least the sampling period T, the review window length, the indicator whitelist, the action type whitelist, and the aggregation function set; when the strategy parameters or thresholds are hot-updated, the graph_profile_revision (graph configuration revision number) and budget_revision (budget revision number) are written into the audit chain to ensure that changes in the graph scope and slight changes are synchronously traceable, and to support the restoration of the corresponding graph scope when rolling back the strategy, avoiding incomparability and misjudgment caused by graph scope drift.

[0038] Preferably, when the budget execution interception layer detects that a video memory allocation request from container i will result in video memory usage... Exceeding the video memory budget When an error occurs, the interface returns an error code with the container handle and request context, and the memory daemon triggers the corresponding degradation action, such as reducing the batch size, switching to a low-memory operator, or triggering garbage collection. When the budget allows but global constraints will be violated, the daemon first shrinks the budget of low-priority containers according to priority, and then allows high-priority containers to continue to allocate, thereby transforming the OOM risk into a controllable policy decision.

[0039] To ensure the idempotency and verifiability of budget changes, this invention returns the actual used memory budget applied_budget and budget revision budget_revision in the memory budget issuance / update operation SET_BUDGET(handle,B_i). The memory usage and failure event reading operation GET_STATS(handle) returns the revision status stats_revision and timestamp. The daemon associates budget_revision, action sequence, and the error context of GET_LAST_ERROR(handle) that supports auditable fault attribution in the audit log, thereby associating which budget issuance caused which allocation failure or latency degradation to the same chain of evidence, avoiding the unreproducible nature caused by simply piecing together logs.

[0040] This invention employs a container identifier mapping layer and a memory statistics and quota interface to attribute and bind memory usage to container objects, forming container-level memory budget semantics. A memory daemon process is used to construct a closed-loop control system for collection, evaluation, execution, and verification. A policy engine introduces priority, threshold, hysteresis, and safety reservation to ensure stable operation of edge devices under strong boundary resource constraints. An executor combines budget shaping, cache reclamation, and inference parameter adjustment into an executable action sequence and provides rollback / degradation mechanisms to reduce runtime risks. Through the cooperation of these modules, memory stability under concurrent inference in multi-container edge ASICs is improved.

[0041] like Figure 2 As shown, the present invention also provides a method for container memory budget control and dynamic optimization, comprising the following steps: S1: Container Registration and Mapping: After the container starts, the monitoring and decision engine obtains relevant metrics of each container's video memory usage during container runtime. These metrics include: container list, container_id, cgroup path, and main process PID. Then, it calls the video memory statistics and quota interface to register the container, obtain the container handle (container_handle), and establish a mapping. S2: The strategy engine calculates the initial budget for each container based on the total device memory, security reserve, and container priority, and then sends it to the executor, which then issues and records a snapshot. S3: The executor issues the initial budget of each container to the memory statistics and limit interface, and then performs memory scheduling through the ASIC edge device scheduler and hardware resources; S4: The monitoring and decision engine collects the memory usage, allocation failure count and critical latency of each container at a preset period, calculates the memory pressure index, and forms statistics within a time window. When the memory pressure index exceeds the memory pressure threshold or an allocation failure occurs, the strategy engine is triggered. S5: The strategy engine outputs an executable sequence of optimized actions to the executor, the sequence of optimized actions including cache reclamation and inference parameter adjustment items; S6: The executor performs budget shaping according to the action sequence, triggers inference runtime cache reclamation, and adaptively adjusts inference parameters, such as adjusting batch size, KV Cache upper limit or precision mode, to reduce peak memory usage, while calculating performance scores. S7: The monitoring and auditing module reads the optimized memory usage indicators and the new video memory pressure index in the review window. If the pressure is relieved, it returns to step S4 to proceed to the next round of data collection. If the pressure does not improve or affects the Service Level Agreement (SLA), it rolls back to the previous strategy or performs a downgrade and issues an alarm. In this invention, the Service Level Agreement (SLA) is reflected in business assurance requirements such as inference latency, throughput, and availability.

[0042] Furthermore, the present invention also includes a graph generation and strategy calibration step: periodically aggregating the indicators and action records of each container, generating a performance graph of container-level time series and associated views through the performance graph module, and calibrating the designed threshold and weight parameters based on the samples in the performance graph.

[0043] As an embodiment of the present invention, the strategy engine in this example makes only two simple adjustments within a fixed time window based on samples of triggering causes, actions, and effects: one is to appropriately relax / tighten the triggering conditions, and the other is to increase / decrease the priority of related actions to reduce invalid triggers and jitter. All adjustments are recorded in versions and can be rolled back.

[0044] As a preferred embodiment of this invention, in order to make the policy calibration a feasible and closed-loop process, the method for the policy engine of this invention to calibrate the various thresholds and weight parameters of the design based on samples in the performance graph is as follows: (1) Set the sample points S={M} of each container in the performance graph. i B i F i L i Action, E i P i} is the input, where i is the index of the container, and M is the index of the container. i B represents the video memory usage of the i-th container. i For the memory budget of the i-th container, F i L is the allocation failure count for the i-th container. i Let E be the critical latency of the i-th container, and Action be the optimization action. i P is the score for the optimization effect of the i-th container. i Let be the memory pressure index of the i-th container; (2) Observe the memory usage trend by sliding average within a fixed time window. If a certain type of action generally brings improvement, then appropriately relax the trigger conditions or increase the priority of the action; if it is ineffective or causes jitter, then tighten the trigger conditions or reduce the priority of the action. Bind all adjustments and calibrations, action records and calibration effects to facilitate auditing and rollback.

[0045] Furthermore, in step S2, the initial budget is calculated as follows: based on the total device memory... Deducting safety margin Then, according to the priority of container i and historical baseline The available budget is allocated proportionally, and the result is truncated at... The interval, where, , Let $i$ be the minimum and maximum budget allowed for container $i$, respectively.

[0046] Furthermore, during the operational phase, the target budget Without violating the overall constraints, the system makes small, automatic adjustments based on the difference between the current memory usage and the target budget, allocation failure signals, and memory stress levels, ensuring the result is always limited to... To avoid excessive shaking.

[0047] Preferably, to quantify the effectiveness of optimization, this invention defines an optimization effect score, which takes a value of 0–1 and is used for panel presentation and strategy calibration. The method for obtaining the optimization effect score in this example is as follows: The system collects three metrics: whether the optimization still exceeds the budget, whether there are still allocation failures, and whether the latency exceeds the threshold. These three metrics are aggregated and normalized according to preset weights to obtain a score. A higher score indicates more effective optimization, while a persistently low score reduces the trigger priority of this type of action or adjusts its weight.

[0048] To facilitate the quantification of memory pressure, this invention adopts a three-level judgment of "low / medium / high", with the following rules: (1) When the video memory usage within the observation window reaches a preset threshold relative to the budget, it is considered to satisfy one condition; (2) When a memory allocation failure event occurs within the observation window, it is recorded as satisfying one condition; (3) When the critical reasoning delay within the observation window exceeds the preset delay threshold, it is recorded as satisfying one of the conditions.

[0049] A score of 2 or higher is considered high pressure; a score of 1 is considered medium pressure; and a score of 0 is considered low pressure. These thresholds are configured uniformly by the policy engine as policy parameters and are recorded with each version to support auditing and rollback.

[0050] Preferably, the strategy engine in this example introduces a hysteresis window and a review window in the determination of the memory pressure index, and restricts budget updates to... Within the interval; this example also sets a global constraint formula:

[0051] in, For the number of concurrent containers, This provides a safety reserve for drivers, systems, and emergency buffers. When constraints are violated or stress increases, the policy engine can perform budget shrinkage and parameter degradation on low-priority containers according to priority to ensure the stable operation of critical containers.

[0052] As can be seen from the above, the present invention has the following innovative points: 1. In this example, the container identifier mapping layer maps container_id / cgroup / pid to container_handle, and the budget execution interception layer associates each allocation with container_handle at the driver interface or user-mode allocator. This enables the read memory usage and failure event GET_STATS, and supports auditable fault attribution GET_LAST_ERROR to output error context on a container-by-container basis, avoiding the situation where only global memory monitoring can be performed without the ability to enforce constraints.

[0053] 2. In this example, under the circumstances of budget enforcement differences and stability risks in different vendor interfaces, a multi-mode strategy such as rejection / blocking / degradation is adopted. The reasons for budget overrun and the request stage are fed back to the memory daemon process through a unified error code and context structure. The memory daemon process prioritizes triggering recoverable actions and only performs rejection or container-level recovery when the risk continues to rise.

[0054] 3. The strategy engine introduces hysteresis windows and review windows in the decision-making process, and restricts budget updates to a set threshold range. When global constraint recycling is triggered, low-priority containers are shrunk in a hierarchical manner according to priority to suppress jitter caused by frequent parameter tuning.

[0055] 4. When the confidence level is insufficient, the memory model construction module will fall back to the conservative upper bound and bind the model version with budget_revision and write it into the audit chain, so that the budget allocation and downgrade actions have an interpretable basis and can be rolled back and reproduced.

[0056] Compared with the prior art, the beneficial effects of the present invention are:

[0057] 1. Form container-level hard boundaries for video memory: By providing control semantics for container objects, video memory statistics, and video memory limits on the driver side or runtime side, each container has a configurable video memory budget limit, reducing mutual interference; 2. Alleviate memory pressure without interrupting the container: By monitoring memory pressure and dynamically optimizing the executor, the model inference parameters and caching strategy can be adjusted in a controllable manner during runtime to avoid frequent restarts and device resets; 3. Balancing edge service priority and stability: Dynamically adjust the video memory budget based on service priority and risk threshold to ensure critical tasks are guaranteed when video memory is scarce, while limiting the peak usage of low-priority tasks. 4. Improve observability, auditability and reproducibility: Video memory usage is attributed to container objects and audit logs are generated, supporting the tracing of OOM risks and optimization actions, which facilitates the operation and maintenance reproduction and strategy iteration of batch devices; 5. Improve the accuracy and interpretability of strategy decisions: By constructing a performance graph, memory usage, latency, allocation failure, and optimization actions are associated with the same container object and timeline, enabling strategy weights and thresholds to be calibrated based on data, avoiding misjudgments and over-adjustments caused by relying solely on experience.

[0058] The specific embodiments described above are preferred embodiments of the present invention and are not intended to limit the specific scope of the present invention. The scope of the present invention includes, but is not limited to, these specific embodiments. All equivalent changes made in accordance with the present invention are within the protection scope of the present invention.

Claims

1. A container memory budget control and dynamic optimization system for ASIC edge devices, characterized in that, include: Memory Statistics and Limit Interface: Set at the input of the ASIC edge device scheduler, used for container registration, and memory control is performed on the container as the object; Container Identifier Mapping Layer: Used to map runtime information provided by the container runtime to container handles that can be recognized by the driver / runtime library, enabling memory statistics and budget control on a container-by-container basis; Monitoring and Decision Engine: Used to periodically collect memory usage-related indicators for each container, calculate the memory pressure index based on the memory usage-related indicators, and trigger the strategy engine action if the memory pressure index exceeds the memory pressure threshold. Strategy Engine: Used to define memory pressure thresholds and optimize action execution rules, and output executable optimized action sequences to the executor; Executor: Located between the strategy engine and the memory statistics and limit interface, it is used to perform budget shaping, trigger inference runtime cache reclamation, and adaptively adjust inference parameters according to the optimized action sequence; Monitoring and Auditing Module: Connected to the output of the monitoring and decision engine and the memory statistics and limit interface respectively, it is used to collect memory indicators, optimize actions and effect verification results, and issue alerts for memory pressure events and risks caused by insufficient memory.

2. The container memory budget control and dynamic optimization system for ASIC edge devices according to claim 1, characterized in that: It also includes a performance graph module: used to perform correlation analysis between memory usage-related indicators, memory pressure index and optimization action sequence, and to provide calibration data for the strategy engine.

3. The container memory budget control and dynamic optimization system for ASIC edge devices according to claim 2, characterized in that: It also includes a memory usage model building module: used to generate a memory usage model based on container runtime data and output peak budget suggestions and phased budget suggestions.

4. The container memory budget control and dynamic optimization system for ASIC edge devices according to claim 3, characterized in that: It also includes a budget execution interception layer: deployed at the driver interface or user-space memory allocator, used to perform budget verification on the container's memory requests, and to intercept requests when the requests cause the memory to exceed the budget.

5. The container memory budget control and dynamic optimization system for ASIC edge devices according to claim 4, characterized in that: After the budget execution interception layer intercepts the request, it returns an error code with the container handle and request context, and the monitoring and decision engine triggers the corresponding degradation action. When the budget is allowed but the global constraints will be violated, the monitoring and decision engine shrinks the budget of low-priority containers according to priority, and then allows high-priority containers to continue to allocate.

6. A method for container memory budget control and dynamic optimization, implemented based on the container memory budget control and dynamic optimization system for ASIC edge devices as described in any one of claims 1-5, characterized in that, Includes the following steps: S1: After the container starts, the monitoring and decision engine obtains relevant indicators of each container's video memory usage, obtains the container handle, and establishes a mapping during container runtime; S2: The policy engine calculates the initial budget for each container based on the total device memory, security reserve, and container priority, and then sends it to the executor; S3: The executor issues the initial budget of each container to the memory statistics and limit interface, and then performs memory scheduling through the ASIC edge device scheduler and hardware resources; S4: The monitoring and decision engine collects the memory usage, allocation failure count and critical latency of each container at a preset period, calculates the memory pressure index, and forms statistics within a time window. When the memory pressure index exceeds the memory pressure threshold or an allocation failure occurs, the strategy engine is triggered. S5: The strategy engine outputs an executable sequence of optimized actions to the executor, the sequence of optimized actions including cache reclamation and inference parameter adjustment items; S6: The executor performs budget shaping according to the action sequence, triggers inference runtime cache reclamation, adaptively adjusts inference parameters, and calculates performance scores. S7: The monitoring and auditing module reads the optimized memory usage indicators and the new video memory pressure index in the review window. If the pressure is relieved, it returns to step S4. If the pressure is not improved or affects the service level agreement (SLA), it rolls back to the previous policy or performs a downgrade and issues an alarm.

7. The container memory budget control and dynamic optimization method according to claim 6, characterized in that: It also includes graph generation and strategy calibration steps: periodically aggregating the metrics and action records of each container, generating a performance graph of container-level time series and associated views through the performance graph module, and calibrating the various threshold and weight parameters of the design based on the samples in the performance graph.

8. The container memory budget control and dynamic optimization method according to claim 7, characterized in that: The method by which the strategy engine calibrates the various threshold and weight parameters of the design based on samples in the performance graph is as follows: (1) Set the sample points S={M} of each container in the performance graph. i B i F i L i Action, E i P i } is the input, where i is the index of the container, and M is the index of the container. i B represents the video memory usage of the i-th container. i For the memory budget of the i-th container, F i L is the allocation failure count for the i-th container. i Let E be the critical latency of the i-th container, and Action be the optimization action. i P is the score for the optimization effect of the i-th container. i Let be the memory pressure index of the i-th container; (2) Observe the memory usage trend by sliding average within a fixed time window. If a certain type of action generally brings improvement, then appropriately relax the trigger conditions or increase the priority of the action; if it is ineffective or causes jitter, then tighten the trigger conditions or reduce the priority of the action, and bind all adjustments with calibration, action recording and calibration effects.

9. The container memory budget control and dynamic optimization method according to claim 6, characterized in that: In step S2, the initial budget is calculated as follows: based on the total amount of device video memory... Deducting safety margin Then, according to the priority of container i and historical baseline The available budget is allocated proportionally, and the result is truncated at... The interval, where, , Let $i$ be the minimum and maximum budget allowed for container $i$, respectively.

10. The container memory budget control and dynamic optimization method according to claim 6, characterized in that: During the operational phase, the target budget Without violating the overall constraints, the system makes small, automatic adjustments based on the difference between the current memory usage and the target budget, allocation failure signals, and memory stress levels, ensuring the result is always limited to... Define an optimization performance score for panel presentation and strategy calibration. The strategy engine introduces a hysteresis window and a review window in the determination of memory stress index, and restricts budget updates to... Within the interval; when global constraint reclamation is triggered, lower-priority containers are stratified and contracted according to priority. The global constraint formula is: in, For the number of concurrent containers, This is a safety reserve for drivers, systems, and emergency buffers.

Images