Elastic inference engine systems, methods, devices, and storage media

Abstract

The application provides an elastic reasoning engine system, method, device and storage medium, wherein the method comprises: a global decision module configured to receive a reasoning task and output an initial configuration set by using multi-objective optimization; a configuration execution module configured to configure the initial configuration set in a deep learning model; each module is called to update in real time; an expansion pruning module configured to perceive the resolution of an input image, adjust the depth layer of the deep learning model by using a resolution mapping function, and adjust the width layer according to the device computing power evaluation; a resource monitoring module configured to predict the resource trend by using an exponential moving average algorithm, adjust the accuracy of a feature map based on the resource trend, and adjust the configuration of a sub-module to be activated in the deep learning model structure according to the input image complexity and the current resource margin; and a multi-precision decoding module configured to store a feature map by using an original precision, generate a corresponding memory view under different precision requirements, and recover the memory view after reasoning is completed.

Description

Technical Field

[0001] This invention relates to the field of deep learning model inference deployment technology, and in particular to an elastic inference engine system, method, device and storage medium. Background Technology

[0002] With the widespread application of deep learning in fields such as computer vision and natural language processing, deploying trained models on edge devices has become a mainstream trend. However, edge devices are highly limited in terms of computing power, memory, and power consumption, which places strict requirements on model inference latency and resource consumption.

[0003] In related technologies, current deep learning model inference deployments typically employ static model structures, meaning that the network depth, width, and computational precision are fixed after model training. During edge device deployment, offline model compression techniques such as pruning and quantization are usually used to reduce model size, or the model is uniformly quantized before deployment, such as converting FP32 to INT8. These solutions maintain the model structure unchanged during inference, using the same computational path regardless of the input image resolution or the device's current computational load. Furthermore, existing multi-task inference systems typically train independent models for each output requirement or use single-precision feature maps for decoding all tasks.

[0004] Currently, deep learning model deployment for inference suffers from the following drawbacks: Static models cannot dynamically adjust the network structure based on input resolution and real-time device computing load, leading to wasted computational resources on low-load devices and excessively high inference latency on high-load devices, failing to achieve optimal resource utilization. The inference engine does not monitor memory usage and latency during inference and cannot trigger on-demand activation of sub-modules based on real-time resource status, easily causing memory overflow or inference timeouts. The same feature map typically only supports single-precision decoding, failing to adapt to the varying precision requirements of different output tasks, resulting in either excessive precision wasting computational resources or insufficient precision impacting task performance. When deployed on edge devices, balancing latency and memory usage is difficult, making it impossible to meet the resource constraints of edge devices while ensuring inference quality.

[0005] Based on the above analysis of the development status of this technology field, existing technologies lack solutions that can dynamically adjust the network depth and width in real time according to the input resolution and device computing power, continuously monitor memory and latency during inference and trigger sub-modules to activate as needed, and support multi-precision decoding of the same feature map to adapt to different output requirements. Summary of the Invention

[0006] The purpose of this invention is to provide a flexible inference engine system, method, device, and storage medium, aiming to solve the above-mentioned problems in the prior art.

[0007] According to a first aspect of the present invention, a flexible inference engine system is provided, comprising: The global decision-making module receives inference tasks and uses multi-objective optimization to output an initial configuration set; The configuration execution module is used to configure the initial configuration set in the deep learning model to start the inference process; it calls the extended pruning module, resource monitoring module, and multi-precision decoding module to update in real time. An extended pruning module is used to sense the resolution of the input image, adjust the number of depth layers of the deep learning model using a resolution mapping function, and adjust the number of width layers based on the device's computing power. The resource monitoring module is used to predict resource trends using the exponential moving average algorithm, adjust the accuracy of the feature map based on the resource trends, and adjust the configuration of the sub-modules that need to be activated in the deep learning model structure according to the complexity of the input image and the current resource reserves. The multi-precision decoding module is used to store feature maps with the original precision, generate corresponding memory views under different precision requirements, and reclaim the memory views after inference is completed.

[0008] According to a second aspect of the present invention, a flexible inference engine method is provided, comprising: The global decision module receives inference tasks and uses multi-objective optimization to output an initial configuration set. The initial configuration set is configured in the deep learning model through the configuration execution module to start the execution inference process; the extended pruning module, resource monitoring module and multi-precision decoding module are called to update in real time; The resolution of the input image is perceived by extending the pruning module, the number of depth layers of the deep learning model is adjusted by using a resolution mapping function, and the number of width layers is adjusted according to the device's computing power. The resource monitoring module uses an exponential moving average algorithm to predict resource trends, adjusts the accuracy of feature maps based on resource trends, and adjusts the configuration of sub-modules that need to be activated in the deep learning model structure according to the complexity of the input image and the current resource availability. The multi-precision decoding module uses the original precision to store feature maps, generates corresponding memory views under different precision requirements, and retrieves the memory views after inference is completed.

[0009] According to a third aspect of the present invention, an electronic device is provided, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the elastic inference engine method provided in the second aspect of the present disclosure.

[0010] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, on which an implementation program for information transmission is stored, which, when executed by a processor, implements the steps of the elastic inference engine method provided in the second aspect of the present disclosure.

[0011] The technical solution provided by this invention includes the following beneficial effects: the initial configuration set output by the global decision module provides unified basic parameters for the subsequent operation of each module, ensuring that the deep learning model can smoothly start inference; through a dynamic expansion pruning mechanism, the model adjusts the network structure in real time according to the input resolution and device computing power, avoiding resource waste on low-load devices and performance bottlenecks on high-load devices, with measured latency reduced by 30% and memory usage reduced by 40%; through continuous resource monitoring and on-demand activation mechanism of sub-modules during inference, the risk of memory overflow and latency timeout can be responded to in a timely manner, automatically triggering degradation strategies to ensure stable operation of inference tasks on resource-constrained edge devices; through a multi-precision decoding mechanism of the same feature map, appropriate computational precision can be allocated according to the precision requirements of different output tasks, avoiding computational waste caused by uniform high precision and performance degradation caused by uniform low precision, achieving the best balance between precision and speed. Overall, it provides a complete end-to-end solution, enabling the same model to adapt to various edge devices and application scenarios, reducing the workload of optimizing models separately for different devices, and involving deep integration of multiple technical fields such as resource monitoring, dynamic scheduling, and multi-precision computation, with a wide range of applications.

[0012] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in one or more embodiments of this specification or in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 This is a schematic diagram of the elastic inference engine system according to an embodiment of the present invention; Figure 2 This is a schematic diagram of dynamic extended pruning according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the inference process resources and submodule activation in an embodiment of the present invention; Figure 4 This is a schematic diagram of multi-precision decoding according to an embodiment of the present invention; Figure 5This is a schematic diagram of the end-to-end decision-making process according to an embodiment of the present invention; Figure 6 This is a flowchart of the elastic inference engine method according to an embodiment of the present invention; Figure 7 This is a schematic diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0015] To enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the technical solutions in one or more embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of this document.

[0016] System Implementation Examples According to embodiments of the present invention, a flexible inference engine system is provided. Figure 1 This is a schematic diagram of the elastic inference engine system according to an embodiment of the present invention, such as... Figure 1 As shown, the elastic inference engine system according to an embodiment of the present invention specifically includes: The global decision module 10 is used to receive inference tasks and output an initial configuration set using multi-objective optimization. Global decision module 10 is similar to an end-to-end inference decision-making mechanism, specifically including: The state joint module is used to collect data features and device operating status in parallel. After normalizing the data features and device operating status respectively, the modules are concatenated to obtain a joint state vector. In this embodiment of the invention, the data features include the effective resolution R, edge gradient density G, and high-frequency energy E calculated based on the image's dimensions, forming a feature vector. The system collects CPU utilization (c), GPU / NPU utilization (g), available memory (m), and power consumption constraints through system-level APIs to construct a device state vector. ; right R , C and D Z-score normalization is performed on each dimension. ,in, This indicates the data to be processed. and The historical mean and standard deviation of the corresponding dimensions are represented respectively, and the normalized features are concatenated into a joint state vector. .

[0017] The multi-objective global optimization module is used to input the joint state vector into the multi-objective optimization model and output the initial configuration set. The cost function of the multi-objective optimization model is expressed by Equation 1. : Formula 1; in, This represents the initial configuration set to be decided. Indicates the initial depth mask. Indicates the initial width mask. This represents the initial submodule configuration activation vector. This represents the initial precision routing table. Indicates the video memory threshold. Indicates the delay threshold. Indicates the precision switching point; This represents the inference delay based on historical regression model predictions. This indicates the estimated video memory usage based on the configuration lookup table. This indicates the accuracy of the reference model estimated based on configuration lookup table. Indicates the weighting coefficient; In the embodiments of the present invention It can be configured according to actual needs; The precision routing table specifies which numerical precision the network layer should use; the initial submodule configuration activation vector defines the functional modules to be started, such as activation base modules, texture analysis, semantic understanding, etc.

[0018] Executing Formula 1 causes the decision-maker to heuristically search or search via a lightweight network in a discrete allocation space to find the optimal allocation method. Minimum current optimal configuration The contents of the initial configuration set to be decided can be configured according to actual needs, such as adding, reducing or changing them. Some parameters are threshold settings required by other modules. These settings do not need to be changed in subsequent applications. Some parameters are initial settings to facilitate normal inference startup, such as depth mask and width mask. Subsequently, each module will dynamically optimize them by executing its own logic.

[0019] The configuration execution module 12 is used to configure the initial configuration set in the deep learning model to start the execution inference process; and to call the extended pruning module, resource monitoring module and multi-precision decoding module to update in real time. Configuration execution module 12 specifically includes: The hot reloading and startup module is used to parse the initial configuration set, i.e., to parse... Based on the initial depth mask and initial width mask, the network structure is constructed using the dynamic computation graph builder DCG Builder, and the activation direction is configured through the initial submodule. The corresponding register is activated, the initial precision routing table and precision switching point are sent to the multi-precision decoding module, the video memory threshold and latency threshold are sent to the resource monitoring module, and the inference process is started.

[0020] The engine utilizes memory pool reuse and operator graph fusion technology to complete the configuration and application within a single inference lifecycle, achieving seamless end-to-end switching.

[0021] Configuration execution module 12 also includes: The call control module is used to call the extension pruning module during the deep learning model's inference task receiving stage; to call the resource monitoring module during the feature extraction, feature fusion, and decoding output stages; and to call the multi-precision decoding module during the feature extraction stage. When dynamic updates are not invoked, the initial configuration set or original settings are used for execution.

[0022] Preferably, the model formed by the graph builder can be drawn as a graph, the execution process of the graph can be monitored to automatically trigger the corresponding module, and information can be automatically shared through the status table without manual intervention.

[0023] The extended pruning module 14 is used to sense the resolution of the input image, adjust the number of depth layers of the deep learning model using a resolution mapping function, and adjust the number of width layers based on the device's computing power. The extended pruning module 14 specifically includes: The dynamic mapping module is used to extract the length and width dimensions of the input image and calculate the effective resolution. ,in, Indicates the length of the input image. The width of the input image is used, and the number of depth layers is adjusted using the resolution mapping function in Formula 2: Formula 2; in, Indicates the target depth layer number. Indicates mapping computation, Indicates the baseline network depth. This represents the depth adjustment coefficient, with a preferred range of 0.5-2.0. This represents the training reference resolution, which is the recommended reference value. This indicates the preset lower limit for pruning. Indicates the preset expansion limit; Before keeping Each network layer generates a binary depth mask vector. ; Preferably, in this embodiment of the invention, the layer importance index table LII, which was pre-computed during the offline training phase of the model, is read, and the layers are retained in descending order of LII values. Each network layer has an LII, which is a layer importance quantization table pre-computed during the offline training phase.

[0024] The capability assessment module is used to obtain the device's computing capability status parameters through system APIs such as NVML, sysfs, or HAL. It uses Formula 3 to calculate a multidimensional load score as the computing capability assessment result. Formula 3; in, This represents a multidimensional load score. Indicates CPU utilization. Indicates CPU / NPU utilization. Indicates available memory. Represents total memory. Indicates the weighting coefficient; Based on the scaling factor of the multidimensional load score mapping , The scaling factor for the width layer can be customized by specifying the hierarchical mapping. When completely idle A value of 1.0 retains 100% of the channels for full-speed calculation; Adjust the channels of the convolutional layer and extract the batch normalization (BN) layer for each channel. Parameter absolute value or weight L1 norm, retaining top-ranked parameters in descending order of sensitivity. Channels, generate binary width mask vectors Sensitivity here refers to the degree of influence of a channel on the model output.

[0025] The instantiation module is updated to encapsulate binary depth mask vectors and binary width mask vectors into a structured configuration file, such as JSON / Protobuf format. The structured configuration file is input into a custom dynamic computation graph builder (DCGBuilder). DDG parses the mask vectors and performs pruning or expansion of redundant nodes in the model graph at runtime. The optimized computation subgraph is generated as an executable file through JIT subgraph compilation. The zero-copy memory pool is called to allocate tensor space according to the mask size to complete the adjustment, realizing hot loading of the network structure. There is no need to reread the model weight file or restart the inference process throughout the process.

[0026] Figure 2 This is a schematic diagram of dynamic expansion pruning according to an embodiment of the present invention, as shown below. Figure 2 As shown, the dynamic expansion / pruning workflow is demonstrated, which includes input resolution detection, resolution threshold comparison, selection of network layer number; device computing power assessment, computing power scoring, adjustment of channel width; generation of structure configuration, and real-time instantiation of network.

[0027] The resource monitoring module 16 is used to predict resource trends using the exponential moving average algorithm, adjust the accuracy of the feature map based on the resource trends, and adjust the configuration of the sub-modules that need to be activated in the deep learning model structure according to the complexity of the input image and the current resource balance. Resource monitoring module 16 specifically includes: Trend prediction module, used for fixed sampling period Collect data on video memory usage and computation latency, and use the Exponential Moving Average (EMA) algorithm (Formula 4) to predict resource trends: Formula 4; in, express Predicted video memory usage at any given time. express Real-time memory usage sampling value, express The calculation delay value for time prediction, express Calculate the delay sample value at any time. Indicates the attenuation coefficient; Time series are used to predict the future. By tracking resource trends at specific time steps, proactive intervention can be implemented to avoid hard interruptions.

[0028] The key decision module is used to determine the current situation. Greater than the video memory threshold or Greater than the delay threshold When this happens, a feature map accuracy degradation strategy is triggered. Formula 5 is used to determine the information entropy and spatial variance of the current intermediate feature map, combined with the task contribution weight of the offline annotation. Calculate non-critical scores : Formula 5; in, The information entropy of the feature map, This represents the maximum information entropy across all feature maps. This represents the spatial variance of the infographic. This represents the maximum spatial variance across all feature maps. Indicates the weight of task contribution; Non-critical scores below the dynamic threshold The feature map is marked as non-critical, and the execution accuracy is downgraded, for example, FP32→FP16 / INT8. Conventional methods such as releasing memory or releasing resource reserves can also be used. The submodule activation module is used to calculate the edge gradient density of the input image. With frequency domain high-frequency energy The computational complexity of the image is ,in, The weighting coefficients represent the image complexity and the current resource availability. Input a lightweight decision model or a pre-compiled activation matrix, and update the output submodule's configuration activation vector. ,in, Indicates activation function Submodules; Edge gradient density It is the result of binarization and summation after Sobel filtering; The pre-compiled activation matrix defines the threshold determination method, for example, when and At that time, only the basic submodules (such as...) are activated. ),when or hour, and Using a matrix threshold, extended sub-modules such as texture analysis and semantic understanding are activated according to a preset topology combination, avoiding full computational overhead.

[0029] Figure 3 This is a schematic diagram of the inference process resources and submodule activation in an embodiment of the present invention, as shown below. Figure 3 As shown, the real-time monitoring module collects video memory / latency data, determines threshold triggers, and triggers degradation strategies; the sub-module activation controller determines the activation of the edge detection / texture analysis / semantic understanding sub-modules based on the input complexity.

[0030] The multi-precision decoding module 18 is used to store feature maps with the original precision, generate corresponding memory views under different precision requirements, and reclaim the memory views after inference is completed.

[0031] The multi-precision decoding module 18 specifically includes: The multi-precision view generation module is used to retain the feature map of the original FP32 precision. When a request for different precisions of the decoding branch is received, the hardware-accelerated dynamic type conversion operator is called to generate the memory view of the corresponding precision. FP16 or INT8 memory views are generated as needed. The life cycle of the memory view is managed by a counting mechanism. The memory view is automatically reclaimed after inference is completed to avoid the expansion of video memory caused by multiple physical copies. The task routing mode module is used to configure precision-sensitive labels for inference tasks through offline verification and calibration. It updates the precision routing table based on the precision-sensitive labels and the current memory usage level. The memory usage level is, for example, how much resource is left. In other words, within the range that the system can meet, the precision-sensitive labels are used to match a threshold range in the precision routing table and update the precision request. In the embodiments of the present invention, the final precision routing table allocates FP16 / FP32 paths for tasks with TSI ≥ 0.8 (such as fine segmentation and key point detection); allocates INT8 paths for tasks with TSI ≤ 0.5 (such as coarse classification and target existence judgment); and uses FP16 as the intermediate value.

[0032] Intra-layer adaptive mode module, used within a single decoding network, to adjust the feature map resolution if... The depth of the deep learning model is greater than the resolution threshold and less than or equal to the precision switching point. If the condition is met, a preset low-precision request is used; otherwise, a preset high-precision request is used. The precision switching point can use the basic configuration and can be optimized through precision-delay Pareto front offline search. The task routing mode is equivalent to determining the overall precision of the feature map based on the current task type, while the intra-layer adaptive mode is equivalent to adaptive adjustment within each layer, belonging to layer adaptive hybrid precision decoding.

[0033] Figure 4 This is a schematic diagram of multi-precision decoding according to an embodiment of the present invention, as shown below. Figure 4 The diagram illustrates the process of feature extraction, multi-precision feature map storage (FP32 / FP16 / INT8), decoding precision selector, different precision decoding for different task branches, and mixed precision decoding output.

[0034] The above technical solutions of the embodiments of the present invention will be illustrated with reference to the following accompanying drawings.

[0035] Figure 5 This is a schematic diagram of the end-to-end decision-making process according to an embodiment of the present invention; as shown... Figure 5 As shown, the complete decision-making process includes receiving inference requests, analyzing input features and device status, making global optimization decisions, generating configuration schemes, executing engine loading of structure / configuration monitoring / activation modules / setting precision, and executing inference.

[0036] In summary, addressing the existing problems, this invention presents an elastic inference engine system. The global decision module outputs an initial configuration set, providing unified basic parameters for subsequent modules, ensuring the smooth inference of the deep learning model. A dynamic expansion and pruning mechanism allows the model to adjust its network structure in real time based on input resolution and device computing power, avoiding resource waste on low-load devices and performance bottlenecks on high-load devices. Actual measurements show a 30% reduction in latency and a 40% reduction in GPU memory usage. Continuous resource monitoring and on-demand activation of sub-modules during inference promptly respond to risks of GPU memory overflow and latency timeouts, automatically triggering degradation strategies to ensure stable operation of inference tasks on resource-constrained edge devices. A multi-precision decoding mechanism using the same feature map allocates appropriate computational precision according to the precision requirements of different output tasks, avoiding computational waste from uniform high precision and performance degradation from uniform low precision, achieving an optimal balance between precision and speed. Overall, it provides a complete end-to-end solution, enabling the same model to adapt to various edge devices and application scenarios, reducing the workload of optimizing models individually for different devices. It involves deep integration of multiple technical fields such as resource monitoring, dynamic scheduling, and multi-precision computation, making it widely applicable.

[0037] Method Implementation Examples According to embodiments of the present invention, a method for an elastic inference engine is provided. Figure 6 This is a flowchart of the elastic inference engine method according to an embodiment of the present invention, such as... Figure 6 As shown, the elastic inference engine method according to an embodiment of the present invention specifically includes: In step S610, the inference task is received through the global decision module, and the initial configuration set is output using multi-objective optimization. In step S620, the initial configuration set is configured in the deep learning model through the configuration execution module to start the execution inference process; the extended pruning module, resource monitoring module and multi-precision decoding module are called to update in real time; In step S630, the resolution of the input image is perceived by the extended pruning module, the number of depth layers of the deep learning model is adjusted by the resolution mapping function, and the number of width layers is adjusted according to the device's computing power. In step S640, the resource monitoring module uses the exponential moving average algorithm to predict resource trends, adjusts the accuracy of the feature map based on the resource trends, and adjusts the configuration of the sub-modules that need to be activated in the deep learning model structure according to the complexity of the input image and the current resource reserves. In step S650, the original precision storage feature map is used by the multi-precision decoding module to generate corresponding memory views under different precision requirements, and the memory views are retrieved after inference is completed.

[0038] In summary, addressing the existing problems, this invention presents an elastic inference engine method. The global decision module outputs an initial configuration set, providing unified basic parameters for subsequent modules, ensuring the deep learning model can smoothly initiate inference. A dynamic expansion and pruning mechanism allows the model to adjust its network structure in real time based on input resolution and device computing power, avoiding resource waste on low-load devices and performance bottlenecks on high-load devices. Actual measurements show a 30% reduction in latency and a 40% reduction in GPU memory usage. Continuous resource monitoring and on-demand activation of sub-modules during inference promptly respond to risks of GPU memory overflow and latency timeouts, automatically triggering degradation strategies to ensure stable operation of inference tasks on resource-constrained edge devices. A multi-precision decoding mechanism using the same feature map allocates appropriate computational precision according to the precision requirements of different output tasks, avoiding computational waste from uniform high precision and performance degradation from uniform low precision, achieving an optimal balance between precision and speed. Overall, this provides a complete end-to-end solution, enabling the same model to adapt to various edge devices and application scenarios, reducing the workload of optimizing models individually for different devices. It involves deep integration of multiple technical fields such as resource monitoring, dynamic scheduling, and multi-precision computation, making it widely applicable.

[0039] Electronic device examples Figure 7 This is a schematic diagram of an electronic device according to an embodiment of the present invention. The electronic device 700 may include at least one processor 710 and a memory 720. The processor 710 can execute instructions stored in the memory 720. The processor 710 is communicatively connected to the memory 720 via a data bus. In addition to the memory 720, the processor 710 can also be communicatively connected to an input device 730, an output device 740, and a communication device 750 via the data bus.

[0040] The processor 710 can be any conventional processor, such as a commercially available CPU. The processor may also include graphics processing units (GPUs), field-programmable gate arrays (FPGAs), systems on chips (SoCs), application-specific integrated circuits (ASICs), or combinations thereof.

[0041] The memory 720 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0042] In this embodiment of the present disclosure, the memory 720 stores executable instructions, and the processor 710 can read the executable instructions from the memory 720 and execute the instructions to implement all or part of the steps of any of the elastic inference engine methods in the exemplary embodiments described above.

[0043] Computer-readable storage medium embodiments In addition to the methods and systems described above, exemplary embodiments of this disclosure may also be a computer program product or a computer-readable storage medium storing the computer program product, the computer product including computer program instructions that can be executed by a processor to implement all or part of the steps described in any of the elastic inference engine methods in the exemplary embodiments described above.

[0044] Computer program products can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. Programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages, and scripting languages ​​(e.g., Python). The program code can be executed entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0045] Computer-readable storage media may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example,, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media include: static random access memory (SRAM) having one or more electrically connected wires; electrically erasable programmable read-only memory (EEPROM); erasable programmable read-only memory (EPROM); programmable read-only memory (PROM); read-only memory (ROM); magnetic storage; flash memory; magnetic disk or optical disk; or any suitable combination thereof.

[0046] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A flexible inference engine system, characterized in that, include: The global decision-making module receives inference tasks and uses multi-objective optimization to output an initial configuration set; The configuration execution module is used to configure the initial configuration set in the deep learning model to start the inference process; The extended pruning module, resource monitoring module, and multi-precision decoding module are invoked for real-time updates. An extended pruning module is used to sense the resolution of the input image, adjust the number of depth layers of the deep learning model using a resolution mapping function, and adjust the number of width layers based on the device's computing power. The resource monitoring module is used to predict resource trends using an exponential moving average algorithm, adjust the accuracy of the feature map based on the resource trends, and adjust the configuration of the sub-modules to be activated in the deep learning model structure according to the complexity of the input image and the current resource availability. The multi-precision decoding module is used to store feature maps with the original precision, generate corresponding memory views under different precision requirements, and retrieve the memory views after inference is completed.

2. The system according to claim 1, characterized in that, The global decision-making module specifically includes: The state joint module is used to collect data features and device operating status in parallel, and after normalizing the data features and the device operating status respectively, they are concatenated to obtain a joint state vector. The multi-objective global optimization module is used to input the joint state vector into the multi-objective optimization model and output the initial configuration set. Equation 1 represents the cost function of the multi-objective optimization model. : Official 1; in, This represents the initial configuration set to be decided. Indicates the initial depth mask. Indicates the initial width mask. This represents the initial submodule configuration activation vector. This represents the initial precision routing table. Indicates the video memory threshold. Indicates the delay threshold. Indicates the precision switching point; This represents the inference delay based on historical regression model predictions. This indicates the estimated video memory usage based on the configuration lookup table. This indicates the accuracy of the reference model estimated based on configuration lookup table. This represents the weighting coefficient.

3. The system according to claim 2, characterized in that, The configuration execution module specifically includes: The hot loading and startup module is used to parse the initial configuration set, construct the network structure according to the initial depth mask and the initial width mask, activate the corresponding register through the initial sub-module configuration activation vector, send the initial precision routing table and the precision switching point to the multi-precision decoding module, send the video memory threshold and the latency threshold to the resource monitoring module, and start the execution inference process.

4. The system according to claim 1, characterized in that, The configuration execution module specifically includes: The control module is invoked to call the extension pruning module during the inference task receiving stage of the deep learning model; to call the resource monitoring module during the feature extraction, feature fusion and decoding output stages; and to call the multi-precision decoding module during the feature extraction stage.

5. The system according to claim 1, characterized in that, The extended pruning module specifically includes: The dynamic mapping module is used to extract the length and width dimensions of the input image and calculate the effective resolution. ,in, Indicates the length of the input image. The width of the input image is used, and the number of depth layers is adjusted using the resolution mapping function in Formula 2: Official 2; in, Indicates the target depth layer number. Indicates mapping computation, Indicates the baseline network depth. This represents the depth adjustment coefficient. Indicates the training reference resolution. This indicates the preset lower limit for pruning. Indicates the preset expansion limit; Before keeping Each network layer generates a binary depth mask vector. ; The capability assessment module is used to obtain the device's computing capability status parameters through the system API and calculate a multi-dimensional load score using Formula 3 as the computing capability assessment result. Official 3; in, This represents a multidimensional load score. Indicates CPU utilization. Indicates CPU / NPU utilization. Indicates available memory. Represents total memory. Indicates the weighting coefficient; Based on the multidimensional load score, the scaling factor for the number of width layers is determined, the convolutional layer channels are adjusted, and a binary width mask vector is generated. ; The instantiation module is updated to encapsulate the binary depth mask vector and the binary width mask vector into a structured configuration file. The structured configuration file is then input into a custom dynamic computation graph builder, which performs pruning or expansion at runtime. An executable file is generated through JIT subgraph compilation, and the adjustment is completed by calling the zero-copy memory pool to allocate tensor space.

6. The system according to claim 1, characterized in that, The resource monitoring module specifically includes: The trend prediction module is used to collect memory usage and computation latency at a fixed sampling period, and uses the exponential moving average algorithm in Formula 4 to predict resource trends: Official 4; in, express Predicted video memory usage at any given time. express Real-time memory usage sampling value, express The calculation delay value for time prediction, express Calculate the delay sample value at any time. Indicates the attenuation coefficient; The key decision module is used to determine the current situation. Greater than the video memory threshold or Greater than the delay threshold When this occurs, a feature map accuracy degradation strategy is triggered, and non-critical scores are calculated using Formula 5 based on the information entropy and spatial variance of the feature map. : Official 5; in, The information entropy of the feature map, This represents the maximum information entropy across all feature maps. This represents the spatial variance of the infographic. This represents the maximum spatial variance across all feature maps. Indicates the weight of task contribution; The non-critical score is below the dynamic threshold. The feature maps are marked as non-critical, resulting in a degraded execution accuracy. The submodule activation module is used to calculate the edge gradient density of the input image. With frequency domain high-frequency energy The computational complexity of the image is ,in, The weighting coefficient represents the sum of the image complexity and the current resource availability. Input a lightweight decision model or a pre-compiled activation matrix, and update the output submodule's configuration activation vector. ,in, Indicates activation function Submodules of.

7. The system according to claim 1, characterized in that, The multi-precision decoding module specifically includes: The multi-precision view generation module is used to retain the feature map of the original precision. When a request for different precisions of the decoding branch is received, hardware acceleration is called to generate the memory view of the corresponding precision. The life cycle of the memory view is managed by a counting mechanism and the memory view is automatically reclaimed after inference is completed. The task routing mode module is used to configure precision-sensitive labels for inference tasks through offline verification and calibration, update the precision routing table according to the precision-sensitive labels and the current video memory usage level, and update the precision request by matching the threshold range in the precision routing table with the precision-sensitive labels. Intra-layer adaptive mode module, used within a single decoding network, to adjust the feature map resolution if... If the resolution is higher than the threshold and the number of layers in the deep learning model is less than or equal to the precision switching point, a preset low-precision request is used; otherwise, a preset high-precision request is used.

8. A flexible inference engine method, used in the flexible inference engine system according to any one of claims 1 to 7, characterized in that, include: The global decision module receives inference tasks and uses multi-objective optimization to output an initial configuration set. The initial configuration set is configured in the deep learning model by configuring the execution module to start the execution inference process; The extended pruning module, resource monitoring module, and multi-precision decoding module are invoked for real-time updates. The resolution of the input image is perceived by extending the pruning module, the number of depth layers of the deep learning model is adjusted by using a resolution mapping function, and the number of width layers is adjusted according to the device's computing power. The resource monitoring module uses an exponential moving average algorithm to predict resource trends, adjusts the accuracy of the feature map based on the resource trends, and adjusts the configuration of the sub-modules that need to be activated in the deep learning model structure according to the complexity of the input image and the current resource balance. The multi-precision decoding module uses the original precision storage feature map to generate corresponding memory views under different precision requirements, and retrieves the memory view after inference is completed.

9. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the elastic inference engine method as described in claim 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores an implementation program for information transmission, which, when executed by a processor, implements the steps of the elastic inference engine method as described in claim 8.

Images