A scene-adaptive model deployment method, system, device and storage medium

Abstract

The present disclosure relates to a scene-adaptive model deployment method, system, device and storage medium. The method comprises: performing combined compression on an initial model to obtain a compressed model; wherein the combined compression comprises a structured pruning operation based on a learnable mask mechanism and a mixed precision quantization operation based on the output change rate under input perturbation; performing frequency domain adaptive enhancement on the intermediate layer feature map in the compressed model to obtain a multi-scale feature map of the compressed model; generating a candidate model configuration set adapted to the multi-scale feature map based on the hardware constraints of a target device; wherein the candidate model configuration comprises at least part of the configurations of quantization precision, hierarchical inference exit point and multi-hypothesis reconstruction order; and in a deployment runtime, selecting an optimal model configuration from the candidate model configuration set according to real-time input data features and real-time resource states of the target device, so that the compressed model performs inference.

Description

Technical Field

[0001] This disclosure relates to the field of model compression technology, and more specifically, to a scenario-adaptive model deployment method, system, device, and storage medium. Background Technology

[0002] While the performance of large language models and deep neural networks continues to break through, their high computational and storage costs severely restrict their deployment in edge devices, mobile terminals, and IoT scenarios. Model compression techniques (including pruning, quantization, knowledge distillation, and low-rank decomposition) address resource constraints by reducing the number of model parameters and computational complexity. However, compressed models face a key challenge in practical deployment: static compression strategies cannot adapt to dynamically changing application scenarios—when the input distribution shifts (such as changes in lighting, network fluctuations, or dynamic changes in device resources), the performance of the compressed model will significantly degrade. Summary of the Invention

[0003] This disclosure provides a scenario-adaptive model deployment method, system, device, and storage medium.

[0004] According to a first aspect of this disclosure, a scene-adaptive model deployment method is provided, the method comprising:

[0005] The initial model is subjected to combined compression to obtain a compressed model; wherein, the combined compression includes structured pruning operation based on a learnable mask mechanism and mixed precision quantization operation based on the output change rate under input perturbation;

[0006] Frequency domain adaptive enhancement is performed on the intermediate layer feature map in the compression model to obtain the multi-scale feature map of the compression model.

[0007] Based on the hardware constraints of the target device, a set of candidate model configurations adapted to the multi-scale feature map is generated; wherein, the candidate model configuration includes at least some of the configurations of quantization accuracy, hierarchical inference exit point, and multi-hypothesis reconstruction order;

[0008] During deployment and runtime, the optimal model configuration is selected from the candidate model configuration set based on the characteristics of real-time input data and the real-time resource status of the target device, so that the compressed model can perform inference.

[0009] Optionally, the learnable masking mechanism includes a Gumbel-Sigmoid relaxation mechanism;

[0010] The mixed-precision quantization operation based on the output change rate under input perturbation includes:

[0011] Based on the mapping relationship between the output change rate and the quantization bit width, the quantization bit width of each layer in the initial model is allocated.

[0012] Optionally, frequency domain adaptive enhancement is performed on the intermediate layer feature maps in the compressed model to obtain multi-scale feature maps of the compressed model, including:

[0013] Discrete wavelet transform is performed on the intermediate layer feature map in the compression model to obtain low-frequency sub-bands and high-frequency sub-bands; wherein, the high-frequency sub-bands include horizontal high-frequency sub-bands, vertical high-frequency sub-bands and diagonal high-frequency sub-bands;

[0014] The low-frequency subband is compressed with a first intensity and the high-frequency subband is compressed with a second intensity to obtain compressed low-frequency subband and high-frequency subband; wherein the first intensity is higher than the second intensity.

[0015] The compressed low-frequency subband and high-frequency subband are subjected to inverse wavelet transform to obtain the multi-scale feature map of the compressed model.

[0016] Optionally, inverse wavelet transform is performed on the compressed low-frequency subband and high-frequency subband to obtain the multi-scale feature map of the compressed model, including:

[0017] The compressed low-frequency subband and high-frequency subband are subjected to inverse wavelet transform, and the high-frequency subband is compensated for information based on the original high-frequency residual to obtain the multi-scale feature map of the compressed model.

[0018] Optionally, the target device hardware constraints include computational capability constraints, memory bandwidth constraints, and cache hierarchy constraints; based on the target device hardware constraints, a set of candidate model configurations adapted to the multi-scale feature map is generated, including:

[0019] A neural architecture search is performed based on the target device hardware constraints and a multi-objective Pareto optimal solution strategy to generate a set of candidate model configurations adapted to the multi-scale feature map; wherein, the multi-objective includes network architecture, pruning ratio and quantization bit width of each layer.

[0020] Optionally, based on the characteristics of real-time input data and the real-time resource status of the target device, the optimal model configuration is selected from the candidate model configuration set, including:

[0021] When the real-time resource status of the target device is higher than the preset threshold, the real-time configuration fusion mode is obtained based on the complexity of the real-time input data characteristics and the mapping relationship between the complexity and the configuration fusion mode.

[0022] Based on the real-time configuration fusion mode and the real-time resource status of the target device, select multiple candidate model configurations from the candidate model configuration set;

[0023] The configurations of the multiple candidate models are fused based on the learnable weight matrix to obtain the optimal model configuration.

[0024] Optionally, the method further includes: when the real-time resource status of the target device is lower than a preset threshold, selecting a candidate model configuration from the candidate model configuration set as the optimal model configuration based on the real-time resource status of the target device.

[0025] According to a second aspect of this disclosure, a scenario-adaptive model deployment system is provided, comprising:

[0026] The combined compression module is used to perform combined compression on the initial model to obtain a compressed model; wherein, the combined compression includes structured pruning operation based on a learnable mask mechanism and mixed precision quantization operation based on the output change rate under input perturbation;

[0027] The frequency domain enhancement module is used to perform frequency domain adaptive enhancement on the intermediate layer feature map in the compressed model to obtain the multi-scale feature map of the compressed model.

[0028] The configuration optimization module is used to generate a set of candidate model configurations adapted to the multi-scale feature map based on the hardware constraints of the target device; wherein, the candidate model configuration includes at least some of the configurations of quantization accuracy, hierarchical inference exit point, and multi-hypothesis reconstruction order;

[0029] An adaptive deployment module is used to select the optimal model configuration from the candidate model configuration set during deployment runtime based on the characteristics of real-time input data and the real-time resource status of the target device, so that the compressed model can perform inference.

[0030] According to a third aspect of this disclosure, a model deployment device is provided, comprising:

[0031] At least one processor; and

[0032] Memory that is communicatively connected to at least one processor;

[0033] The memory stores instructions that can be executed by at least one processor, such that the at least one processor is able to perform any of the methods in the first aspect.

[0034] According to a fourth aspect of this disclosure, a non-transitory computer-readable storage medium is provided, the non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform any of the methods of the first aspect.

[0035] This disclosure significantly improves the inference performance and resource utilization of edge devices through model compression and dynamic adaptation techniques. First, offline combined compression effectively reduces the number of model parameters and computational complexity, decreasing storage and energy consumption requirements. Frequency domain enhancement enhances the model's ability to capture key information through multi-scale feature maps, compensating for accuracy loss caused by compression. Parameter optimization generates a candidate model configuration set based on hardware constraints, providing optimization space for real-time scheduling. Online dynamic scheduling flexibly switches configurations based on input features and device status, achieving a dynamic balance between accuracy and efficiency. This solution maintains model lightweighting while improving inference flexibility and scenario adaptability through hardware and software co-optimization, increasing the efficiency of model deployment in resource-constrained environments.

[0036] Other features and advantages of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0037] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments of the present disclosure and, together with their description, serve to explain the principles of the present disclosure.

[0038] Figure 1 This is a flowchart illustrating a scenario-adaptive model deployment method provided in an embodiment of this disclosure.

[0039] Figure 2 This is a schematic diagram of the structure of a scenario-adaptive model deployment system provided in an embodiment of this disclosure.

[0040] Figure 3 This is a schematic diagram of the structure of a model deployment device provided in an embodiment of this disclosure. Detailed Implementation

[0041] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present disclosure.

[0042] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0043] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0044] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0045] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0046] Figure 1 This illustration shows a flowchart of a scene-adaptive model deployment method provided by an embodiment of this disclosure. This disclosure provides a scene-adaptive model deployment method, such as... Figure 1 As shown, the method includes steps S110 to S140.

[0047] Step S110: Perform combinatorial compression on the initial model to obtain a compressed model; wherein, combinatorial compression includes structured pruning operation based on a learnable mask mechanism and mixed precision quantization operation based on the output change rate under input perturbation.

[0048] In this embodiment, multiple compression techniques are used to compress the initial model in a coordinated manner by combining structured pruning and mixed precision quantization of the model.

[0049] The structured pruning employs a learnable mask mechanism, where each weight corresponds to an independent, learnable Bernoulli variable. The learnable mask adaptively learns the tendency to retain and remove different weights during training. Additionally, a global sparse regularizer can be used to control the overall proportion of weights retained by the model. The global sparse regularizer calculates the L2 distance between the current density and the target density and adds it as a regularization term to the loss function to ensure that the preset compression ratio requirement is met.

[0050] Mixed precision quantization assigns different quantization bit widths to each layer based on the importance of each layer in the model. Layers with high importance are assigned higher bit widths, while layers with low importance are assigned lower bit widths.

[0051] In some examples, learnable masking mechanisms include Gumbel-Sigmoid relaxation, a technique that transforms a binary discrete decision process into a continuously differentiable surrogate operation. Its core is to approximate hard binary sampling behavior by introducing Gumbel noise and temperature parameter control, while maintaining gradient backpropagation, so that the model can optimize the binary decision variables through standard backpropagation.

[0052] The formula for generating the learnable mask m corresponding to each weight through Gumbel-Sigmoid sampling is as follows:

[0053] m=σ((logα+ G) / τ),

[0054] Where α is a learnable parameter, G is Gumbel noise, and τ is the temperature coefficient.

[0055] The Gumbel-Sigmoid relaxation mechanism can maintain the end-to-end differentiability of the learnable mask, allowing pruning decisions to be jointly optimized with other compression strategies.

[0056] In this example, mixed-precision quantization, based on the output rate of change under input perturbation, includes allocating the quantization bit width of each layer in the initial model based on the mapping relationship between the output rate of change and the quantization bit width.

[0057] The importance of each model layer is measured by calculating the rate of change of the output of each layer's features under input perturbation. The formula for calculating the rate of change of output is as follows:

[0058] ,

[0059] Layers with low rates of change are more robust. For robust layers, such as FFN layers, a lower bit width (such as INT4) is used to maximize the compression ratio while maintaining overall accuracy.

[0060] In this example, an asymmetric quantization strategy can also be introduced. For activation values ​​with asymmetric distributions (such as non-negative values ​​after ReLU), the offset encoding is set as follows:

[0061] ,

[0062] Where z is the zero-point offset.

[0063] In this example, sensitive layers can also be identified using an outlier suppression algorithm. The Fused Iter-smooth reverse smoothing algorithm is used to identify layers with large outliers in the activation values ​​and to mark such layers as sensitive layers, retaining a higher precision bit width (such as FP16 or INT16) than that of non-sensitive layers.

[0064] Step S120: Perform frequency domain adaptive enhancement on the intermediate layer feature map in the compressed model to obtain the multi-scale feature map of the compressed model.

[0065] In this embodiment, the intermediate layer feature map in the compression model is decomposed in the frequency domain, each frequency domain is enhanced with adaptive intensity, and then each enhanced frequency domain is restored to the intermediate layer feature map to obtain the multi-scale feature map of the compression model.

[0066] In some examples, the frequency-domain adaptive enhancement operation includes: performing a discrete wavelet transform on the intermediate layer feature map of the compressed model to obtain low-frequency and high-frequency subbands, wherein the high-frequency subbands include horizontal, vertical, and diagonal high-frequency subbands. Then, the low-frequency subbands are compressed with a first intensity, and the high-frequency subbands are compressed with a second intensity, resulting in compressed low-frequency and high-frequency subbands, where the first intensity is higher than the second intensity. Finally, an inverse wavelet transform is performed on the compressed low-frequency and high-frequency subbands to obtain the multi-scale feature map of the compressed model.

[0067] In this example, after performing discrete wavelet transform on the intermediate layer feature map, it is decomposed into four sub-bands: LL (low-frequency sub-band), LH (horizontal high-frequency sub-band), HL (vertical high-frequency sub-band), and HH (diagonal high-frequency sub-band).

[0068] The low-frequency subband contains the smooth structure and global semantic information of the image, which contributes greatly to the task output and is sensitive to compression errors. Therefore, the first intensity is used for compression, such as low-rank decomposition combined with aggressive quantization, while retaining sufficient accuracy.

[0069] The high-frequency subband contains detailed information such as edges and textures, and has a higher tolerance for compression errors. Therefore, a second intensity lower than the first intensity is used for compression, such as INT8 quantization or FP16 retention.

[0070] In actual deployment, the number of high-frequency subbands to be retained can be dynamically selected based on the computing power of the target device. For low-end devices (such as MCUs and Raspberry Pis), only low-frequency subbands are retained to meet real-time requirements; for high-end devices (such as GPU edge servers), low-frequency subbands and some or all of the high-frequency subbands are retained to obtain the best reconstruction quality.

[0071] In some examples, inverse wavelet transform is performed on the compressed low-frequency subband and high-frequency subband, and information compensation is performed on the high-frequency subband based on the original high-frequency residual to obtain the multi-scale feature map of the compressed model.

[0072] The original high-frequency residuals include high-frequency details (such as edges and textures) that were not captured by the reconstructed feature map in the original feature map. Since they are mainly concentrated in the high-frequency region, they can be directly used to compensate for the missing components of the high-frequency subband in the wavelet transform.

[0073] Step S130: Based on the hardware constraints of the target device, generate a set of candidate model configurations adapted to the multi-scale feature map; wherein, the candidate model configuration includes at least some of the configurations of quantization accuracy, hierarchical inference exit point and multi-hypothesis reconstruction order.

[0074] In this embodiment, a hardware-aware neural architecture search is employed. The target device's hardware characteristics serve as constraints in the search space. An architecture adapted to the multi-scale feature map is searched among multiple optimization objectives to ensure that the searched architecture can run efficiently on the target hardware. Then, based on the searched architecture, an optimization algorithm generates a set of candidate model configurations matched to the multi-scale feature map. The optimization objectives may include model accuracy (e.g., Top-1 classification accuracy), inference latency (measured latency on the target hardware), and model size (number of parameters or storage volume).

[0075] In some examples, the target device hardware constraints include computational power constraints, memory bandwidth constraints, and cache hierarchy constraints. A neural architecture search is performed based on the target device hardware constraints and a multi-objective Pareto optimal solution strategy to generate a set of candidate model configurations adapted to the multi-scale feature map; wherein, the multi-objectives include network architecture, pruning ratio, and quantization bit width of each layer.

[0076] In this example, the model configuration architecture search is performed using the target device's hardware computing power, memory bandwidth, and cache hierarchy as search space constraints. A multi-objective evolutionary optimization algorithm (such as NSGA-II or MOEA / D) is employed to search for a non-dominated optimal solution set on the Pareto front. A quantized-aware accuracy prediction period can be used instead of a full model evaluation. This quantized-aware accuracy prediction period is a lightweight neural network that takes the architecture encoding and quantized configuration as input and directly outputs the prediction accuracy.

[0077] Each solution on the Pareto front represents a combination of architecture and quantization configuration that cannot improve any objective without affecting other objectives, thus achieving a balance among the objectives. The generated candidate model configuration includes parameters such as pruning masks, quantization bit widths, and convolutional kernel sizes for each layer, and each candidate model configuration satisfies the hardware constraints of the target device.

[0078] In this example, a four-dimensional quantitative evaluation index system can also be constructed to guide the optimization direction of the parameter optimization strategy. The four dimensions include model size, inference latency, accuracy retention rate, and hardware compatibility.

[0079] The model size reflects storage and memory usage, with parameters and storage volume (MB / GB) as sub-indicators, and the optimization objective is to minimize these parameters.

[0080] Inference latency reflects real-time response capability, with single-sample latency (ms) and throughput (tokens / sec) as sub-indicators. The optimization objective is to meet service level agreement constraints (e.g., latency <50ms).

[0081] The accuracy retention rate reflects the impact of compression on task performance. The percentage of task metrics of the compressed model relative to the original model is used as a sub-metric, and the optimization objective is no less than 99%.

[0082] Hardware compatibility reflects the degree of matching with the target hardware, with instruction set support compatibility (such as ARM NEON, Tensor Core) and memory bandwidth utilization as sub-indicators. The optimization goal is full compatibility with a utilization rate of no less than 60%.

[0083] Step S140: During deployment, the optimal model configuration is selected from the candidate model configuration set based on the real-time input data characteristics and the real-time resource status of the target device, so that the compressed model can perform inference.

[0084] During the deployment and operation of the compression model, the resource status of the target device is monitored in real time, and the complexity of the input samples is evaluated. Based on the resource status and the complexity of the input samples, some configurations in quantization accuracy, hierarchical inference exit point, and multi-hypothesis reconstruction order are dynamically adjusted.

[0085] For example, when resources are scarce, the accuracy can be downgraded from INT8 to INT4, while maintaining high accuracy when resources are plentiful; or, simple samples can exit early at the shallow end of the network, while complex samples can be routed to the deeper layers; or, the number of hypotheses K to be fused can be selected based on the input complexity.

[0086] Resource status can include some status data such as CPU utilization, available memory, battery level, and network bandwidth. The complexity of the input samples can be measured and evaluated using methods such as classifier confidence, gradient magnitude variance, or local binary patterns.

[0087] In some examples, when the real-time resource status of the target device is higher than a preset threshold, a real-time configuration fusion mode is obtained based on the complexity of the real-time input data features and the mapping relationship between the complexity and the configuration fusion mode; based on the real-time configuration fusion mode and the real-time resource status of the target device, multiple candidate model configurations are selected from the candidate model configuration set; the multiple candidate model configurations are fused based on a learnable weight matrix to obtain the optimal model configuration.

[0088] In this example, based on the complexity of the real-time input data features, the corresponding configuration fusion mode is determined through the mapping relationship between complexity and configuration fusion mode. The complexity of the input data features can be measured using data features such as pixel variance and edge density. The configuration fusion mode can include the number of fusion hypotheses K and a subset of data from the selected candidate model configuration.

[0089] For example, when the real-time resource status of the target device is higher than a preset threshold, simple inputs (such as smooth backgrounds) use low-order fusion (K=2), while complex inputs (such as high-texture images) use high-order fusion (K=4-6).

[0090] This example uses a learnable weight matrix to fuse multiple candidate models to configure the output of the corresponding version model. The learnable weight matrix is ​​as follows:

[0091] ,

[0092] in, The weights are learnable and are dynamically calculated through an attention mechanism.

[0093] When the real-time resource status of the target device falls below a preset threshold, it automatically downgrades to single-hypothesis reconstruction (K=1), i.e., standard decompression. Based on the real-time resource status of the target device, a candidate model configuration is selected from the candidate model configuration set as the optimal model configuration.

[0094] In some examples, the dynamic adjustment of the hierarchical inference exit point is based solely on a rapid preliminary judgment using low-frequency sub-band features. Since the low-frequency sub-band contains the smooth structure and global semantics of the image, and is small in size and computationally fast after compression, this stage can be completed with extremely low latency. When the confidence level of the preliminary judgment is higher than a first threshold (e.g., 0.95), the system exits directly at the shallow exit and outputs the result.

[0095] When the confidence level is below the first threshold, it indicates that the input sample is relatively complex, and low-frequency information is insufficient to make a reliable judgment. At this point, high-frequency sub-band features are further combined for refined reasoning. High-frequency sub-bands provide detailed information such as edges and textures, and their fusion with low-frequency sub-bands can significantly improve the ability to discriminate complex samples.

[0096] This disclosure significantly improves the inference performance and resource utilization of edge devices through model compression and dynamic adaptation techniques. First, offline combined compression effectively reduces the number of model parameters and computational complexity, decreasing storage and energy consumption requirements. Frequency domain enhancement enhances the model's ability to capture key information through multi-scale feature maps, compensating for accuracy loss caused by compression. Parameter optimization generates a candidate model configuration set based on hardware constraints, providing optimization space for real-time scheduling. Online dynamic scheduling flexibly switches configurations based on input features and device status, achieving a dynamic balance between accuracy and efficiency. This solution maintains model lightweighting while improving inference flexibility and scenario adaptability through hardware and software co-optimization, increasing the efficiency of model deployment in resource-constrained environments.

[0097] Figure 2 This disclosure illustrates a scenario-adaptive model deployment system 200, as provided in an embodiment of the present disclosure. Figure 2As shown, the scene-adaptive model deployment system 200 includes a combined compression module 210, a frequency domain enhancement module 220, a configuration optimization module 230, and an adaptive deployment module 240; wherein:

[0098] The combined compression module 210 is used to perform combined compression on the initial model to obtain a compressed model; wherein, the combined compression includes structured pruning operation based on a learnable mask mechanism and mixed precision quantization operation based on the output change rate under input perturbation.

[0099] The frequency domain enhancement module 220 is used to perform frequency domain adaptive enhancement on the intermediate layer feature map in the compressed model to obtain the multi-scale feature map of the compressed model.

[0100] The configuration optimization module 230 is used to generate a set of candidate model configurations adapted to the multi-scale feature map based on the hardware constraints of the target device; wherein, the candidate model configuration includes at least some of the configurations of quantization accuracy, hierarchical inference exit point and multi-hypothesis reconstruction order.

[0101] The adaptive deployment module 240 is used to select the optimal model configuration from the candidate model configuration set during deployment runtime based on the characteristics of real-time input data and the real-time resource status of the target device, so that the compressed model can perform inference.

[0102] It should be noted that the functions implemented by the combined compression module, frequency domain enhancement module, configuration optimization module and adaptive deployment module can be referred to the description in the foregoing embodiments of this disclosure, and will not be repeated here.

[0103] Figure 3 This is a schematic diagram of the structure of a model deployment device provided in an embodiment of this disclosure. The model deployment device can be... Figure 1 The receiver or pseudo-satellite base station shown can also be any electronic device. For example... Figure 3As shown, the model deployment device 300 may include a memory 310 and a processor 320. The memory 310 may be used to store computer instructions, and the processor 320 may be used to retrieve computer instructions from the memory 310 to execute all or part of the steps of any of the methods in the foregoing embodiments of this disclosure. It should be noted that the processor 320 may include one or more processors to execute instructions, and the memory 310 may also include one or more memories to store computer instructions. In some embodiments, the processor 320 may be used to control the overall operation of the model deployment device 300. For example, the processor 320 may execute instructions to implement all or part of the steps of the methods in any of the foregoing embodiments of this disclosure, thereby enabling one or more operations such as voice communication, data communication, database operation, display control, component control, and multimedia processing. The aforementioned components may be internal components of the model deployment device itself, or external components connected to the model deployment device wirelessly or wiredly. For example, the components may include sensors, cameras, headphones, input / output devices, etc., and the aforementioned multimedia may include one or more of voice, images, video, and text. In some embodiments, the memory 310 may include one or more memories, and the contents stored in different memories may be the same or different. The memory 310 can be configured to store various types of data to support the operation of the model deployment device 300. Examples of such data include instructions for any application or method operating on the model deployment device 300, contact data, phonebook data, messages, pictures, videos, etc. The memory 310 can be implemented by any type of transient or non-transitory storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0104] This disclosure also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the methods in the foregoing embodiments of this disclosure.

[0105] The various embodiments in this disclosure are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device and apparatus embodiments are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0106] The foregoing has described specific embodiments of this disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0107] This disclosure can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this disclosure.

[0108] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0109] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0110] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0111] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0112] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0113] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0114] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It will be known to those skilled in the art that implementation in hardware, implementation in software, and implementation in a combination of software and hardware are equivalent.

[0115] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of this disclosure is defined by the appended claims.

Claims

1. A scenario-adaptive model deployment method, characterized in that, The method includes: The initial model is subjected to combined compression to obtain a compressed model; wherein, the combined compression includes structured pruning operation based on a learnable mask mechanism and mixed precision quantization operation based on the output change rate under input perturbation; Frequency domain adaptive enhancement is performed on the intermediate layer feature map in the compression model to obtain the multi-scale feature map of the compression model. Based on the hardware constraints of the target device, a set of candidate model configurations adapted to the multi-scale feature map is generated; wherein, the candidate model configuration includes at least some of the configurations of quantization accuracy, hierarchical inference exit point, and multi-hypothesis reconstruction order; During deployment and runtime, the optimal model configuration is selected from the candidate model configuration set based on the characteristics of real-time input data and the real-time resource status of the target device, so that the compressed model can perform inference.

2. The method according to claim 1, characterized in that, The learnable masking mechanism includes the Gumbel-Sigmoid relaxation mechanism; The mixed-precision quantization operation based on the output change rate under input perturbation includes: Based on the mapping relationship between the output change rate and the quantization bit width, the quantization bit width of each layer in the initial model is allocated.

3. The method according to claim 1, characterized in that, Frequency domain adaptive enhancement is performed on the intermediate layer feature maps in the compressed model to obtain multi-scale feature maps of the compressed model, including: Discrete wavelet transform is performed on the intermediate layer feature map in the compression model to obtain low-frequency sub-bands and high-frequency sub-bands; wherein, the high-frequency sub-bands include horizontal high-frequency sub-bands, vertical high-frequency sub-bands and diagonal high-frequency sub-bands; The low-frequency subband is compressed with a first intensity and the high-frequency subband is compressed with a second intensity to obtain compressed low-frequency subband and high-frequency subband; wherein the first intensity is higher than the second intensity. The compressed low-frequency subband and high-frequency subband are subjected to inverse wavelet transform to obtain the multi-scale feature map of the compressed model.

4. The method according to claim 3, characterized in that, Performing inverse wavelet transform on the compressed low-frequency subband and high-frequency subband yields the multi-scale feature map of the compressed model, including: The compressed low-frequency subband and high-frequency subband are subjected to inverse wavelet transform, and the high-frequency subband is compensated for information based on the original high-frequency residual to obtain the multi-scale feature map of the compressed model.

5. The method according to claim 1, characterized in that, The target device hardware constraints include computing power constraints, memory bandwidth constraints, and cache hierarchy constraints; based on the target device hardware constraints, a set of candidate model configurations adapted to the multi-scale feature map is generated, including: A neural architecture search is performed based on the target device hardware constraints and a multi-objective Pareto optimal solution strategy to generate a set of candidate model configurations adapted to the multi-scale feature map; wherein, the multi-objective includes network architecture, pruning ratio and quantization bit width of each layer.

6. The method according to claim 1, characterized in that, Based on the characteristics of real-time input data and the real-time resource status of the target device, the optimal model configuration is selected from the candidate model configuration set, including: When the real-time resource status of the target device is higher than the preset threshold, the real-time configuration fusion mode is obtained based on the complexity of the real-time input data characteristics and the mapping relationship between the complexity and the configuration fusion mode. Based on the real-time configuration fusion mode and the real-time resource status of the target device, select multiple candidate model configurations from the candidate model configuration set; The configurations of the multiple candidate models are fused based on the learnable weight matrix to obtain the optimal model configuration.

7. The method according to claim 6, characterized in that, Also includes: When the real-time resource status of the target device is lower than a preset threshold, a candidate model configuration is selected from the candidate model configuration set as the optimal model configuration based on the real-time resource status of the target device.

8. A scenario-adaptive model deployment system, characterized in that, include: The combined compression module is used to perform combined compression on the initial model to obtain a compressed model; wherein, the combined compression includes structured pruning operation based on a learnable mask mechanism and mixed precision quantization operation based on the output change rate under input perturbation; The frequency domain enhancement module is used to perform frequency domain adaptive enhancement on the intermediate layer feature map in the compressed model to obtain the multi-scale feature map of the compressed model. The configuration optimization module is used to generate a set of candidate model configurations adapted to the multi-scale feature map based on the hardware constraints of the target device; wherein, the candidate model configuration includes at least some of the configurations of quantization accuracy, hierarchical inference exit point, and multi-hypothesis reconstruction order; An adaptive deployment module is used to select the optimal model configuration from the candidate model configuration set during deployment runtime based on the characteristics of real-time input data and the real-time resource status of the target device, so that the compressed model can perform inference.

9. A model deployment device, characterized in that, include: At least one processor; as well as A memory that is communicatively connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.

10. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer instructions for causing the computer to perform the method of any one of claims 1-7.

Images