Model training method, vehicle-mounted visual perception platform and computer readable storage medium

Abstract

The embodiment of the invention provides a model training method, a vehicle-mounted visual perception platform and a computer readable storage medium. The method comprises the steps that an initial model and a target function are constructed according to hardware parameters of a target platform, the hardware parameters are used for representing the computing power, energy efficiency and storage limitation of the target platform, and the target function is used for constraining reasoning delay, energy consumption and storage occupation of the initial model; determining a pruning proportion and a first bit width based on an initial weight parameter of the initial model and a target function; discretizing the pruning proportion to obtain a pruning mask, and discretizing the first bit width to obtain a second bit width; adjusting the initial weight parameter based on the pruning mask and the second bit width to obtain a target weight parameter; and generating a target model based on the target weight parameter. According to the method, the technical problems of insufficient calculation power utilization rate, limited energy efficiency improvement and large precision loss of a model optimization method in related technologies are solved.

Description

Technical Field

[0001] This application relates to the field of vehicle technology, and more specifically, to a model training method, an in-vehicle visual perception platform, and a computer-readable storage medium. Background Technology

[0002] In the field of artificial intelligence, deep neural networks (DNNs) have become a core technology for automotive embedded vision perception platforms, playing a crucial role, especially in real-time object detection and tracking. However, there is a significant contradiction between the massive computational and storage requirements of high-performance DNN models and the limited resources of automotive embedded vision perception platforms.

[0003] Currently, model optimization methods are mainly divided into two categories: pruning and quantization. Pruning techniques reduce model complexity, floating-point operations (FLOPs), and the number of parameters by decreasing redundant neurons or connections in the model. Quantization techniques quantize model weights to a lower bit width to reduce storage overhead and bandwidth requirements. However, these methods still face problems such as insufficient computing power utilization, limited energy efficiency improvement, and significant accuracy loss in practical embedded scenarios.

[0004] There is currently no good solution to the above problems. Summary of the Invention

[0005] This application provides a model training method, an in-vehicle visual perception platform, and a computer-readable storage medium to at least solve the technical problems of insufficient computing power utilization, limited energy efficiency improvement, and large accuracy loss in related model optimization methods.

[0006] According to one aspect of the embodiments of this application, a model training method is provided, comprising: constructing an initial model and an objective function based on hardware parameters of a target platform, wherein the hardware parameters represent the computing power, energy efficiency, and storage limitations of the target platform, and the objective function constrains the inference latency, energy consumption, and storage usage of the initial model; determining a pruning ratio and a first bit width based on the initial weight parameters and the objective function of the initial model; discretizing the pruning ratio to obtain a pruning mask, and discretizing the first bit width to obtain a second bit width; adjusting the initial weight parameters based on the pruning mask and the second bit width to obtain target weight parameters; and generating a target model based on the target weight parameters, wherein the target model is used to perform a deep learning task.

[0007] Furthermore, constructing the initial model and objective function based on the hardware parameters of the target platform includes: determining the neural network parameters based on the hardware parameters, wherein the neural network parameters include the number of input channels, the number of output channels, and the dimension of the output feature map; performing micro-benchmark tests on the target platform based on the neural network parameters to determine the performance data of the target platform, wherein the performance data includes runtime latency, power consumption, and peak memory usage; performing multinomial regression processing or neural network fitting processing on the performance data to obtain a prediction function, wherein the prediction function is used to predict the performance of the model on the target platform; and constructing the initial model and objective function based on the prediction function.

[0008] Furthermore, the prediction functions include: a delay prediction function, an energy consumption prediction function, and a memory usage prediction function. The delay prediction function is used to predict the layer operation time, the energy consumption prediction function is used to predict the energy consumption of the layer operation, and the memory usage prediction function is used to predict the peak memory usage of the layer operation.

[0009] Furthermore, determining the pruning ratio and the first bit width based on the initial weight parameters and objective function of the initial model includes: initializing the pruning ratio based on the initial weight parameters and objective function to obtain the initial pruning ratio, and initializing the bit width based on the initial weight parameters and objective function to obtain the initial bit width; iteratively updating the initial pruning ratio based on the differentiable architecture search strategy and the target sampling strategy to obtain the pruning ratio, and iteratively updating the initial bit width based on the differentiable architecture search strategy and the target sampling strategy to obtain the first bit width.

[0010] Furthermore, the process of initializing the pruning ratio based on the initial weight parameters and the objective function to obtain the initial pruning ratio, and initializing the bit width based on the initial weight parameters and the objective function to obtain the initial bit width, includes: calculating the hardware-aware score based on the initial weight parameters and the objective function, wherein the hardware-aware score includes hyperparameter values, which are used to represent the priority of different performance indicators; initializing the pruning ratio based on the hyperparameter values ​​to obtain the initial pruning ratio, and initializing the bit width based on the hyperparameter values ​​to obtain the initial bit width.

[0011] Furthermore, based on the differentiable architecture search strategy and the target sampling strategy, the initial pruning ratio is iteratively updated to obtain the pruning ratio. Similarly, based on the differentiable architecture search strategy and the target sampling strategy, the initial bit width is iteratively updated to obtain the first bit width. This includes iteratively updating the initial pruning ratio based on the differentiable architecture search strategy, the target sampling strategy, and the hyperparameter values ​​to obtain the pruning ratio, and iteratively updating the initial bit width based on the differentiable architecture search strategy, the target sampling strategy, and the hyperparameter values ​​to obtain the first bit width. The hyperparameter values ​​are updated according to the model state and hardware parameters in each iteration.

[0012] Furthermore, the pruning ratio is discretized to obtain a pruning mask, which includes: discretizing the pruning ratio based on a preset threshold to obtain a pruning mask, wherein the preset threshold is used to distinguish between channel retention and pruning.

[0013] Furthermore, the initial weight parameters are adjusted based on the pruning mask and the second bit width to obtain the target weight parameters, including: the initial weight parameters are fine-tuned based on the preset fine-tuning algorithm, the pruning mask, and the second bit width to obtain the target weight parameters.

[0014] According to another aspect of the embodiments of this application, an in-vehicle visual perception platform is also provided, which is equipped with a model trained according to any of the model training methods described above.

[0015] According to another aspect of the embodiments of this application, a model training apparatus is also provided, comprising: a construction module, configured to construct an initial model and an objective function based on hardware parameters of a target platform, wherein the hardware parameters represent the computing power, energy efficiency, and storage limitations of the target platform, and the objective function constrains the inference latency, energy consumption, and storage usage of the initial model; a determination module, configured to determine a pruning ratio and a first bit width based on the initial weight parameters of the initial model and the objective function; a processing module, configured to discretize the pruning ratio to obtain a pruning mask, and discretize the first bit width to obtain a second bit width; an adjustment module, configured to adjust the initial weight parameters based on the pruning mask and the second bit width to obtain target weight parameters; and a generation module, configured to generate a target model based on the target weight parameters, wherein the target model is used to perform a deep learning task.

[0016] According to another aspect of the embodiments of this application, a vehicle is also provided, including: a memory storing an executable program; and a processor for running the program, wherein the program executes the methods in various embodiments of this application when it runs.

[0017] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein, when the executable program is running, it controls the device where the computer-readable storage medium is located to perform the methods of various embodiments of this application.

[0018] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the methods of various embodiments of this application.

[0019] In this embodiment, neural network parameters are determined based on hardware parameters, including the number of input channels, the number of output channels, and the dimension of the output feature map. Then, micro-benchmark tests are performed on the target platform based on these parameters to determine the platform's performance data, including runtime latency, energy consumption, and peak memory usage. The performance data is then processed using multinomial regression or neural network fitting to obtain a prediction function. This prediction function is used to predict the model's performance on the target platform. Finally, an initial model and a target function are constructed based on the prediction function. This achieves a deep integration of intelligent pruning quantization and hardware characteristics, resulting in a significant reduction in computational latency, energy consumption, and storage usage while maintaining task accuracy. It is suitable for deployment in resource-constrained automotive embedded systems, thus solving the technical problems of insufficient computing power utilization, limited energy efficiency improvement, and significant accuracy loss in related model optimization methods. Attached Figure Description

[0020] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0021] Figure 1 This is a flowchart of a model training method according to one embodiment of this application;

[0022] Figure 2 This is another flowchart of a model training method according to one embodiment of this application;

[0023] Figure 3 This is a schematic diagram of the structure of an in-vehicle vision perception platform according to one embodiment of this application;

[0024] Figure 4 This is a structural block diagram of a model training apparatus according to one embodiment of this application. Detailed Implementation

[0025] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0026] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0027] With the widespread adoption of artificial intelligence in mobile terminals, the Internet of Things, and edge computing, the contradiction between the computational complexity and storage overhead of deep neural network models and the computing power, bandwidth, and energy consumption of embedded hardware platforms is becoming increasingly prominent. Traditional pruning or quantization methods mostly address a single aspect, such as reducing the parameter size or the number of quantization bits for weights, but often ignore the differences in latency, energy consumption, and storage characteristics of different layers on actual hardware. This leads to the following problems with traditional methods in real-world embedded deployments:

[0028] Problem 1: Low computational efficiency. Simply relying on FLOPs or parameter reduction does not accurately reflect hardware latency, and the improvement in inference speed after model compression is limited.

[0029] Question 2: Energy consumption and storage bottlenecks are prominent. Embedded devices typically have limited battery capacity. Although the model becomes smaller, energy consumption and bandwidth consumption may not decrease accordingly.

[0030] Question 3: It is difficult to balance accuracy and performance. Although low-bit quantization can significantly reduce model size, it will lead to a large loss of accuracy in key operators, affecting the practical application effect.

[0031] Question 4: Lack of joint optimization strategy. Pruning and quantization are mostly performed independently, lacking global collaborative optimization based on hardware characteristics, often requiring repeated manual debugging.

[0032] While traditional approaches have achieved some success in model pruning, low-bit quantization, and hardware-aware optimization, they still have many shortcomings in practical applications. Specifically:

[0033] In terms of pruning methods, traditional approaches often measure channel importance based on weight size or sparsity, focusing more on reducing the number of model parameters or FLOPs. However, this metric does not fully correspond to the performance on real embedded hardware, resulting in limited improvements in inference latency and energy consumption, and low computational efficiency, even though the model is theoretically "smaller".

[0034] In terms of quantization methods, traditional approaches mainly focus on eight-bit integer quantization (INT8) or mixed-precision quantization. While these methods reduce model storage and bandwidth consumption to some extent, they can still cause significant performance degradation in precision-sensitive key layers. Moreover, pruning and quantization are mostly performed independently, lacking global collaborative optimization, often requiring extensive manual debugging to find a balance between compression ratio and accuracy.

[0035] In hardware-aware optimization, existing work has largely focused on Neural Architecture Search (NAS), adjusting the model architecture through latency prediction functions. However, this approach primarily targets macroscopic structural optimization, lacking support for fine-grained operations such as pruning and quantization. Furthermore, traditional methods typically focus on a single metric, such as latency, rarely considering multi-dimensional constraints like energy consumption and storage. This makes it difficult for traditional methods to meet the comprehensive requirements of embedded devices under conditions of low power consumption, low latency, and limited storage.

[0036] In the joint optimization of pruning and quantization, traditional methods typically treat the pruning ratio and quantization bit width as discretized hyperparameters, determined through exhaustive search or heuristic rules. Since this process is non-differentiable, optimization often requires extensive manual debugging and repeated experiments, resulting in low search efficiency and difficulty in achieving global optimum under multiple objective constraints (accuracy, latency, energy consumption, storage, etc.). Even some studies attempting to introduce joint training still rely on manually defined rules, lacking a unified, differentiable optimization framework, leading to insufficient robustness and generality.

[0037] To address the issues that traditional solutions still face in embedded scenarios, such as insufficient computing power utilization, limited energy efficiency improvement, difficulty in controlling accuracy loss, and lack of joint optimization mechanisms, this application proposes a model training method, specifically a hardware-aware pruning and quantization joint optimization method. By introducing a prediction model of latency, energy consumption, and memory usage, and combining channel-level importance scoring with a mixed-precision quantization strategy, a dynamic trade-off between pruning ratio and quantization accuracy is achieved. This method minimizes the power consumption and inference latency of the embedded platform while ensuring model accuracy.

[0038] The model training method proposed in this application is applicable to in-vehicle embedded visual perception systems, especially to real-time target detection / tracking models (such as detection networks based on DETR / YOLO / RetinaNet) running on in-vehicle systems on a chip / advanced driver assistance system (SoC / ADAS ECU) with limited computing resources, for pedestrian, vehicle, and obstacle recognition and near-far warning.

[0039] According to an embodiment of this application, a method embodiment for model training is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0040] This embodiment provides a model training method. Figure 1 This is a flowchart of a model training method according to one embodiment of this application, such as... Figure 1 As shown, the process includes the following steps:

[0041] Step S11: Construct an initial model and objective function based on the hardware parameters of the target platform. The hardware parameters represent the computing power, energy efficiency, and storage limitations of the target platform, while the objective function constrains the inference latency, energy consumption, and storage usage of the initial model.

[0042] In this embodiment of the application, the target platform can be an automotive embedded platform (such as Jetson Nano). The hardware parameters of the target platform refer to the hardware specifications of the target embedded system (such as automotive SoC or ADAS ECU), including computing power (such as the performance of the central processing unit (CPU) and graphics processing unit (GPU)), energy efficiency (power consumption), and storage limitations (memory and storage size).

[0043] An initial model refers to an unoptimized deep learning model, which typically has high computational complexity, large model size, and high energy consumption.

[0044] The objective function comprehensively considers various indicators such as inference accuracy, latency, energy consumption, and storage usage of the model, and is used to guide the optimization process of the model so that the optimized model can better adapt to the target hardware platform.

[0045] As can be seen, this application first constructs an initial model based on the hardware parameters of the target platform, namely computing power, energy efficiency, and storage limitations. This initial model serves as the basis for subsequent optimizations. However, the initial model may contain too many parameters and computational overhead, making it unsuitable for direct deployment on hardware with limited computing resources. Simultaneously, based on the properties of this initial model and the performance constraints of the target hardware platform, an objective function is defined. The objective function is designed to quantify the model optimization objectives, such as minimizing inference latency and energy consumption, while ensuring that the model's inference accuracy is not lower than a specific threshold. This objective function will guide the subsequent pruning, quantization, and joint optimization process, ensuring that the optimized model is both efficient and accurate.

[0046] Therefore, constructing an initial model ensures that the optimization process begins with a model that meets the specific task requirements, providing a starting point for optimization. Simultaneously, defining the objective function clarifies the direction and constraints of the optimization, providing a quantitative evaluation standard for pruning-quantitative joint optimization, facilitating subsequent automated optimization, and ensuring that the model optimization does not deviate from the needs of practical applications.

[0047] Step S12: Based on the initial weight parameters and objective function of the initial model, determine the pruning ratio and the first bit width.

[0048] In this embodiment of the application, the initial weight parameter The initial neuron connection weights of the model are learned during model training and used for prediction and classification tasks.

[0049] Pruning ratio This refers to the proportion of neurons or connections that are removed from the model in order to reduce computation and model size.

[0050] First bit width This is the initial number of bits used for model weight quantization at the start of the optimization process, typically the original quantization precision of the model.

[0051] As can be seen, this application uses the initial model's weight parameters as a starting point and dynamically adjusts the pruning ratio and quantization bit width by optimizing the objective function. In this stage, the pruning ratio and quantization bit width are initially set as continuous variables to be optimized during the search for differentiable architectures. Through this joint optimization strategy, the model seeks the optimal balance between pruning and quantization to satisfy the constraints on inference latency, energy consumption, and storage footprint in the objective function.

[0052] Therefore, this application can automatically determine the pruning ratio of each layer based on hardware characteristics and task requirements, avoiding the subjectivity and inaccuracy of manual settings. By adjusting the quantization bit width during training, this application can find the optimal balance between model accuracy and storage overhead, thereby improving the model's energy efficiency.

[0053] Step S13: Discretize the pruning ratio to obtain the pruning mask, and discretize the first bit width to obtain the second bit width.

[0054] In this embodiment, the pruning mask It is a binary vector used to indicate which neurons or connections in the model should be kept and which should be deleted.

[0055] Discretization is used to transform continuous optimization variables into discrete values ​​to meet the quantization requirements of actual hardware.

[0056] The second bit width is the final discretized value of the quantization bit width, used to guide the final quantization process of the model.

[0057] As can be seen, since the hardware can only accept a specific bit width when performing quantization operations, discretization is necessary. This application discretizes the pruning ratio obtained through differentiable architecture search to generate a pruning mask. This mask will guide channel-level pruning of the model, ensuring that the pruned model structure is actually runnable on hardware. Simultaneously, the quantization bit width also needs to be converted from continuous optimization results to a value from the set of practically applicable discrete bit widths, i.e., the second bit width.

[0058] Therefore, the generation of the pruning mask ensures that model pruning can be actually executed on hardware, avoiding problems caused by continuous pruning ratios not being directly applicable to hardware. Furthermore, determining the second bit width provides specific guidance for model quantization, ensuring that the final quantization result can be directly run on the target hardware without generating incompatible quantization bit widths.

[0059] Step S14: Adjust the initial weight parameters based on the pruning mask and the second bit width to obtain the target weight parameters.

[0060] In this embodiment of the application, the target weight parameter refers to the model weight that matches the pruning mask and the final quantization bit width after pruning quantization optimization.

[0061] As can be seen, this application uses a pruning mask to adjust the model's weight parameters, removing neurons or connections marked as unnecessary. Simultaneously, based on the determined second bit width, the retained weights are quantized to obtain the target weight parameters. This step reduces the number of model parameters and computational complexity to a level suitable for the target platform, while also reducing the model's storage requirements.

[0062] Therefore, through pruning, a large number of unnecessary parameters are removed, significantly reducing the number of parameters in the model and thus reducing computational and storage costs. Furthermore, quantization based on the second bit width ensures that the model can run efficiently on computationally limited hardware while maintaining high accuracy.

[0063] Step S15: Generate a target model based on the target weight parameters, wherein the target model is used to perform deep learning tasks.

[0064] In this embodiment, the target model refers to a deep learning model that has been pruned and quantized for optimization, making it suitable for efficient operation on the target hardware platform. The target model is used to perform deep learning tasks, such as visual recognition tasks, for pedestrian, vehicle, and obstacle recognition, and near / far distance warnings.

[0065] As can be seen, this application applies optimized weight parameters to the model to generate a target model that has undergone structured pruning and mixed-precision quantization. This target model will be used to perform specific deep learning tasks, such as real-time object detection or tracking. The generated target model is not only computationally efficient, but also significantly improves energy and storage efficiency, making it suitable for deployment in resource-constrained environments such as automotive embedded systems.

[0066] Therefore, the target model, after optimization, can achieve optimal performance under hardware constraints. This is crucial for resource-constrained vehicle systems, as it ensures real-time performance and energy efficiency while maintaining good model accuracy.

[0067] In summary, the pruning technique in this application removes redundant weights, making the model sparser, thereby reducing computational resource consumption and improving computing power utilization. Furthermore, the quantization technique reduces the storage bandwidth of the weights, resulting in fewer memory accesses and lower bandwidth consumption, thus improving overall energy efficiency. In addition, through constraints on the objective function and multi-objective optimization strategies, the model can maintain its core feature representation capabilities during pruning and quantization. Post-processing measures such as fine-tuning further reduce the accuracy loss caused by structural simplification.

[0068] The above steps in this application determine neural network parameters based on hardware parameters, including the number of input channels, the number of output channels, and the dimension of the output feature map. Then, based on these parameters, micro-benchmark tests are performed on the target platform to determine the platform's performance data, including runtime latency, energy consumption, and peak memory usage. Subsequently, multinomial regression or neural network fitting is performed on the performance data to obtain a prediction function. This prediction function is used to predict the model's performance on the target platform. Finally, an initial model and an objective function are constructed based on the prediction function. This achieves a deep integration of intelligent pruning quantization and hardware characteristics, resulting in a significant reduction in computational latency, energy consumption, and storage usage while maintaining task accuracy. It is suitable for deployment in resource-constrained automotive embedded systems, thus solving the technical problems of insufficient computing power utilization, limited energy efficiency improvement, and significant accuracy loss in related model optimization methods.

[0069] Optionally, in step S11, constructing the initial model and objective function based on the hardware parameters of the target platform may include the following execution steps:

[0070] Step S111: Determine the neural network parameters based on the hardware parameters, wherein the neural network parameters include the number of input channels, the number of output channels, and the dimension of the output feature map.

[0071] Step S112: Perform micro-benchmark tests on the target platform based on neural network parameters to determine the performance data of the target platform, including runtime latency, power consumption, and peak memory usage.

[0072] Step S113: Perform multinomial regression or neural network fitting on the performance data to obtain a prediction function, which is used to predict the performance of the model on the target platform.

[0073] Step S114: Construct an initial model and an objective function based on the prediction function.

[0074] In this embodiment of the application, when constructing the initial model and objective function based on the hardware parameters of the target platform, the neural network parameters can be determined based on the hardware parameters.

[0075] The neural network parameters include the number of input channels, the number of output channels, and the dimension of the output feature map; that is, the number of input channels... Number of output channels or kernel size Output feature map dimension It is an important component of the deep learning model structure, directly affecting the model's computational complexity, storage requirements, and performance.

[0076] As can be seen, this application first analyzes the hardware parameters of the target platform and determines the architectural details of the neural network model based on these parameters. Specifically, it rationally selects the number of input channels, the number of output channels, and the dimension of the output feature map in the model according to the computing resources of the target hardware, ensuring that the model is neither too complex to be deployed nor too rudimentary to affect detection or recognition performance.

[0077] Therefore, by setting model parameters guided by hardware parameters, the model structure is matched with the computing power and storage limitations of the target platform, avoiding waste of resources and over-configuration, and providing a solid foundation for the implementation of subsequent optimization strategies.

[0078] Then, micro-benchmark tests are performed on the target platform based on the neural network parameters to determine the performance data of the target platform, including runtime latency, power consumption, and peak memory usage.

[0079] Micro-benchmarking refers to tests run individually on each layer or specific component to measure performance metrics such as runtime latency under different conditions (e.g., different numbers of input channels, output channels, or output feature map dimensions). Energy consumption and peak memory usage .

[0080] As can be seen, this application performs micro-benchmark tests on each layer of the initially designed neural network model on the target platform. By adjusting the number of input channels, the number of output channels, and the dimension of the output feature map, the runtime latency, power consumption, and memory usage under these changes are collected. This step can obtain real performance data of the model on the target hardware, providing a basis for subsequent predictive modeling.

[0081] Therefore, micro-benchmarking can accurately capture the detailed performance of the model running on the target platform, providing real and reliable data support for subsequent hardware-aware performance prediction, and helping to more accurately evaluate and optimize the model's performance on specific hardware.

[0082] Then, the performance data is processed by multinomial regression or neural network fitting to obtain the prediction function, which is used to predict the performance of the model on the target platform.

[0083] Among them, multinomial regression is a statistical prediction model used to fit the nonlinear relationship between data points. It reflects the dependency between input and output variables by constructing a multinomial function.

[0084] Neural network fitting uses a neural network architecture to learn and predict complex patterns in data, approximating the nonlinear mapping between inputs and outputs by training the network.

[0085] As can be seen, this application utilizes the aforementioned collected performance data to establish prediction functions using multinomial regression or neural network techniques. These prediction functions can predict performance indicators such as runtime latency, energy consumption, and peak storage on the target platform based on the input neural network parameters (such as the number of channels, feature map size, etc.).

[0086] Therefore, the establishment of the prediction function frees the model optimization process from dependence on actual hardware operation, and allows for rapid evaluation of the impact of different pruning ratios and quantization bits on performance during training, thereby accelerating the search speed and improving optimization efficiency.

[0087] Finally, an initial model and an objective function are constructed based on the prediction function.

[0088] As can be seen, after obtaining the function that can predict the model's performance on the target platform, this application constructs an initial model based on the prediction function and sets the objective function. Thus, setting the objective function clarifies the model optimization goal, providing a clear direction and quantitative standard for the entire search and optimization process, ensuring that the model maintains a balance with the task's accuracy requirements while pursuing performance improvement.

[0089] Optionally, the prediction function includes: a delay prediction function, an energy consumption prediction function, and a memory usage prediction function. The delay prediction function is used to predict the layer operation time, the energy consumption prediction function is used to predict the energy consumption of the layer operation, and the memory usage prediction function is used to predict the peak memory usage of the layer operation.

[0090] In this embodiment of the application, the delay prediction function The delay prediction function is used to predict the layer operation time, that is, to estimate the layer processing time by analyzing the computing characteristics of the hardware platform, such as processor frequency, cache efficiency, parallel computing capability and memory access time.

[0091] Specifically, the delay prediction function helps to dynamically adjust the pruning ratio and the number of quantization bits during the pruning and quantization process, so as to ensure that the optimized model can meet the real-time requirements of the target platform, help avoid performance degradation caused by over-optimization, and ensure that the model running speed meets the application requirements.

[0092] For example, by running micro-benchmark tests to collect the actual runtime of each layer under different configurations, and using regression analysis (such as multinomial regression or neural networks) to fit a continuous function to predict the runtime under any configuration, a delay prediction function is obtained.

[0093] Energy consumption prediction function The energy consumption prediction function is used to predict the energy consumption of layer operations.

[0094] Specifically, when optimizing the model, the energy consumption prediction function ensures that the model not only runs fast but also has high energy efficiency, which is especially important for battery-powered embedded devices, such as automotive SoCs or ADAS ECUs. By controlling energy consumption, device uptime can be extended and device maintenance costs reduced.

[0095] For example, the energy consumption prediction function is also established through micro-benchmarking and model fitting. In the tests, the actual energy consumption under different configurations is recorded, and then a prediction model is created using a fitting method to estimate the energy consumption of any configuration.

[0096] Memory usage prediction function The memory usage prediction function is used to predict the peak memory usage of layer operations, including the storage requirements of layer weights, biases, input and output feature maps, and any additional intermediate results or buffers.

[0097] Specifically, memory usage prediction functions are particularly important for optimizing models on embedded devices, which typically have limited memory resources. By limiting the model's memory usage, it is possible to ensure that the model runs in a memory-constrained environment while avoiding runtime failures or performance degradation caused by insufficient memory.

[0098] For example, by performing micro-benchmark tests on the target hardware platform to record memory usage under different configurations, a model is fitted using statistical or machine learning methods to predict memory usage under any configuration, resulting in a memory usage prediction function.

[0099] Optionally, in step S12, determining the pruning ratio and the first bit width based on the initial weight parameters and objective function of the initial model may include the following steps:

[0100] Step S121: Initialize the pruning ratio based on the initial weight parameters and the objective function to obtain the initial pruning ratio, and initialize the bit width based on the initial weight parameters and the objective function to obtain the initial bit width.

[0101] Step S122: Based on the differentiable architecture search strategy and the target sampling strategy, iteratively update the initial pruning ratio to obtain the pruning ratio, and based on the differentiable architecture search strategy and the target sampling strategy, iteratively update the initial bit width to obtain the first bit width.

[0102] In this embodiment of the application, when determining the pruning ratio and the first bit width based on the initial weight parameters and objective function of the initial model, the pruning ratio can be initialized based on the initial weight parameters and objective function to obtain the initial pruning ratio, and the bit width can be initialized based on the initial weight parameters and objective function to obtain the initial bit width.

[0103] The initial pruning ratio refers to setting a preliminary pruning ratio based on the initial model's weight parameters and the constraints of the objective function. This is typically based on information such as weight norms and hardware performance predictions, and by correlating the weight parameters with the objective function, it initially determines which channels or weights can be pruned, thereby reducing the number of model parameters.

[0104] Similarly, initializing the bit width refers to initializing the quantization bit width based on the weight parameters and objective function constraints of the initial model. This step reduces the model's storage requirements by determining which layers or channels can tolerate lower precision.

[0105] Therefore, initializing the pruning ratio and bit width provides a starting point for the subsequent optimization process. By considering them in conjunction with the objective function, it is ensured that the optimization direction of the model is oriented towards hardware resource constraints and performance requirements from the outset.

[0106] Then, based on the differentiable architecture search strategy and the target sampling strategy, the initial pruning ratio is iteratively updated to obtain the pruning ratio, and based on the differentiable architecture search strategy and the target sampling strategy, the initial bit width is iteratively updated to obtain the first bit width.

[0107] Differentiable Architecture Search Strategy (DARTS) allows model architecture parameters (such as pruning ratio and quantization accuracy) to participate in training in a continuous manner and be automatically adjusted through backpropagation to achieve the optimization goal.

[0108] Target sampling strategies, such as the Gumbel-Softmax sampling mechanism, dynamically sample and adjust the model architecture parameters during the search process based on the constraints and guidance of the objective function to explore the optimal combination of pruning and quantization.

[0109] Iterative pruning ratio updates refer to using a differentiable architecture search strategy, where the pruning ratio is treated as a continuous variable and dynamically adjusted during training using methods such as gradient descent. This process leverages constraints in the objective function, such as latency and energy budget, and combines the relationship between the pruning ratio and model accuracy to progressively find the optimal pruning strategy.

[0110] Iterative bit width updates are also based on a differentiable architecture search strategy, and the bit width is adjusted as a continuous variable, allowing the model to explore the effects of different quantization accuracies during training. Constraints on the objective function, such as storage footprint and inference accuracy, guide the dynamic optimization of the bit width to find the optimal quantization scheme.

[0111] Thus, through iterative updates, the model can intelligently adjust its structure and quantization accuracy, achieving a dynamic balance between hardware resources and model performance. Furthermore, the combination of differentiable architecture search and target sampling strategies not only accelerates the search process but also increases the likelihood of finding the global optimum, ensuring that the model maintains high accuracy and performance while satisfying hardware constraints.

[0112] Optionally, in step S121, initializing the pruning ratio based on the initial weight parameters and the objective function to obtain the initial pruning ratio, and initializing the bit width based on the initial weight parameters and the objective function to obtain the initial bit width may include the following execution steps:

[0113] Step S1211: Calculate the hardware-aware score based on the initial weight parameters and the objective function. The hardware-aware score includes hyperparameter values, which are used to represent the priority of different performance indicators.

[0114] Step S1212: Initialize the pruning ratio based on the hyperparameter values ​​to obtain the initial pruning ratio, and initialize the bit width based on the hyperparameter values ​​to obtain the initial bit width.

[0115] In this embodiment of the application, when the initial pruning ratio is obtained by initializing the pruning ratio based on the initial weight parameters and the objective function, and the initial bit width is obtained by initializing the bit width based on the initial weight parameters and the objective function, the hardware perception score can be calculated based on the initial weight parameters and the objective function.

[0116] Among them, hardware perception score This refers to a mechanism that scores each layer or channel of a model by quantifying the impact of the model's hardware performance (such as latency and energy consumption variations) and combining this with the importance of weight parameters. This hardware-aware scoring reflects the different preferences of hardware platforms for model structures.

[0117] Hardware perception scoring includes hyperparameter values, hyperparameter values Hyperparameter values ​​are used to indicate the priority of different performance metrics; that is, they are parameters used to adjust the priority of different performance metrics in the hardware-aware scoring function.

[0118] As can be seen, for each layer or channel in the model, a hardware-aware score is calculated based on its initial weight parameters and objective function (i.e., hardware performance constraints). The hardware-aware score function comprehensively considers the channel's weight norm (reflecting the channel's contribution to the model), latency reduction, energy consumption reduction, and memory reduction. Each indicator is assigned a different weight according to its importance in the objective function (i.e., hyperparameter value).

[0119] Therefore, the hardware-aware scoring mechanism transforms abstract hardware performance metrics into intuitive score values, helping to determine which channels or model components are most beneficial for reducing latency, power consumption, and memory. Through hyperparameter values, the priority of performance metrics during the optimization process can be dynamically adjusted, ensuring that model optimization considers both performance and resource consumption, achieving a balance between multiple objectives.

[0120] Then, the pruning ratio is initialized based on the hyperparameter values ​​to obtain the initial pruning ratio, and the bit width is initialized based on the hyperparameter values ​​to obtain the initial bit width.

[0121] The pruning ratio is initialized based on the hyperparameter values, that is, according to the hyperparameter values ​​set in the hardware-aware scoring function, an initial retention probability, i.e., the pruning ratio, is assigned to each layer or channel of the model.

[0122] The quantization bit width is initialized based on the hyperparameter values. Similarly, guided by the hyperparameter values, an initial quantization precision, i.e., the quantization bit width, is set for each layer or channel of the model.

[0123] Therefore, the initialization of the pruning ratio and quantization bit width provides a starting point for the subsequent optimization process, helping to systematically explore the optimization space of model structure and quantization accuracy. The initialization process guided by hyperparameter values ​​ensures that the pruning and quantization schemes are hardware-aware from the beginning, taking into account the characteristics of the hardware platform in advance, and laying the foundation for fine-tuning in subsequent steps.

[0124] Optionally, in step S122, iteratively updating the initial pruning ratio based on the differentiable architecture search strategy and the target sampling strategy to obtain the pruning ratio, and iteratively updating the initial bit width based on the differentiable architecture search strategy and the target sampling strategy to obtain the first bit width may include the following execution steps:

[0125] Step S1221: Based on the differentiable architecture search strategy, target sampling strategy and hyperparameter values, iteratively update the initial pruning ratio to obtain the pruning ratio, and based on the differentiable architecture search strategy, target sampling strategy and hyperparameter values, iteratively update the initial bit width to obtain the first bit width. The hyperparameter values ​​are updated according to the model state and hardware parameters in each iteration.

[0126] In this embodiment of the application, when iteratively updating the initial pruning ratio based on the differentiable architecture search strategy and the target sampling strategy to obtain the pruning ratio, and when iteratively updating the initial bit width based on the differentiable architecture search strategy and the target sampling strategy to obtain the first bit width, the initial pruning ratio can be iteratively updated based on the differentiable architecture search strategy, the target sampling strategy and the hyperparameter value to obtain the pruning ratio.

[0127] During training, the pruning ratio is treated as a continuously relaxed parameter and optimized along with the model weights through backpropagation. In each iteration, the hyperparameter values ​​(including the weight allocation of the pruning ratio) are updated according to the target sampling strategy and the current model state to guide the evolution of the pruning vector, ultimately approximating the optimal pruning ratio.

[0128] Based on the differentiable architecture search strategy, target sampling strategy, and hyperparameter values, the initial bit width is iteratively updated to obtain the first bit width.

[0129] Similarly, the quantization bit width is relaxed to a continuous variable and dynamically adjusted during training using a differentiable quantization function and a target sampling strategy. Likewise, hyperparameter values ​​are updated in each iteration based on the model state and hardware parameters to optimize quantization accuracy, reduce memory usage and energy consumption, while maintaining model accuracy as much as possible.

[0130] Therefore, by treating the pruning ratio and quantization bit width as differentiable continuous variables, they can be dynamically adjusted during model training, achieving hardware-aware joint optimization. This method offers higher efficiency and better optimization results compared to traditional discrete search.

[0131] Furthermore, the iterative updates of the target sampling strategy and hyperparameter values ​​ensure that the optimization direction of the model always remains consistent with the performance constraints of the target hardware platform, and can generate the model structure and quantization configuration most suitable for the platform.

[0132] Furthermore, the automatic updating of hyperparameters during the iteration process reduces the reliance on manual settings, improves the automation of model optimization, and makes the entire model compression and quantization process more efficient and accurate.

[0133] Optionally, in step S13, discretizing the pruning ratio to obtain the pruning mask may include the following steps:

[0134] Step S131: Discretize the pruning ratio based on a preset threshold to obtain a pruning mask, wherein the preset threshold is used to distinguish between channel retention and pruning.

[0135] In this embodiment of the application, when discretizing the pruning ratio to obtain the pruning mask, the pruning ratio can be discretized based on a preset threshold to obtain the pruning mask.

[0136] Among them, the preset threshold A preset threshold is used to distinguish between channel retention and pruning; it's a standard value used during the pruning process to determine whether a channel or weight in a neural network should be retained or deleted. This preset threshold is set based on the pruning ratio dynamically adjusted during training and search, typically to convert continuous pruning probabilities into discrete decisions, i.e., retaining or pruning channels.

[0137] It can be seen that after the search phase ends, the pruning ratio of each channel is compared with a preset threshold τ. If If the probability of retaining a channel is greater than or equal to τ, then the corresponding value in the pruning mask is set to 1, indicating that the channel should be retained. Conversely, if... If the probability of retaining a channel is less than τ, the corresponding value in the pruning mask is set to 0, indicating that the channel will be pruned.

[0138] Therefore, converting continuous pruning probabilities into binary masks clearly indicates which channels should be retained and which should be pruned, making the pruning operation concrete and definite. The application of pruning masks simplifies the execution steps of subsequent pruning algorithms; pruning is achieved simply by multiplying the mask by the original weight matrix, greatly simplifying the model compression process. By setting a preset threshold, the degree of model pruning can be controlled, avoiding performance degradation caused by over-pruning while ensuring efficient model operation on resource-constrained hardware.

[0139] Optionally, in step S14, adjusting the initial weight parameters based on the pruning mask and the second bit width to obtain the target weight parameters may include the following steps:

[0140] Step S141: Based on the preset fine-tuning algorithm, pruning mask, and second bit width, the initial weight parameters are fine-tuned to obtain the target weight parameters.

[0141] In this embodiment, the preset fine-tuning algorithm is an algorithm used to restore model accuracy after model pruning and quantization. Common fine-tuning algorithms include, but are not limited to, low-rank adapter (LoRA), gradient-based fine-tuning, and low-rank matrix factorization.

[0142] As can be seen, this application applies a pruning mask to the initial weight parameters, that is, it performs a masking operation on the weight parameters, retaining the weight values ​​marked as 1 in the pruning mask, while setting the weight values ​​marked as 0 to zero or removing them, thereby directly reducing the number of model parameters and computational complexity. The retained weight parameters are quantized according to the second bit width, that is, the weights are represented using fewer bits, further reducing the model's storage requirements and computational cost. A preset fine-tuning algorithm is used to train the adjusted weight parameters to restore or improve the model's performance as much as possible.

[0143] Therefore, through fine-tuning, the model can recover some or all of the accuracy loss after pruning and quantization, ensuring that the model maintains high accuracy when running on the hardware platform. Determining the pruning mask and quantization bit width significantly reduces the number of model parameters and computational requirements, thereby accelerating the deployment process on embedded devices and improving real-time performance and energy efficiency. Furthermore, the quantized model has a smaller storage footprint and a more lightweight computational process, helping to reduce the memory burden and power consumption of embedded devices and extend their battery life.

[0144] In summary, this application proposes a hardware-aware pruning and quantization joint optimization method, which includes the following steps.

[0145] Step 1: Establish joint optimization objectives.

[0146] Construct a joint optimization objective function that takes into account the model inference accuracy deployed on the target automotive embedded platform, such as Jetson Nano. Model inference delay Model energy consumption The peak memory of the model, M, is given in the following formula.

[0147]

[0148]

[0149]

[0150]

[0151] Where F is the joint optimization objective function on the target in-vehicle embedded platform. For the network weight parameters of the model to be deployed on the platform, This is the pruning ratio vector for each layer. Quantization bit width for each layer, The latency budget for deploying the model on the target platform (generally determined by the real-time nature of the tasks to be performed by the model on the target platform). This is the energy consumption budget (generally the standard energy consumption of the target platform to be deployed). For memory budget (usually the memory size of the target platform).

[0152] Differentiable Architecture Search (DARTS) is employed. and Joint optimization dynamically adjusts the network structure during the search process to find the optimal model architecture to be deployed on the target platform.

[0153] Step 2, target platform hardware-aware performance modeling.

[0154] On the target in-vehicle embedded platform, for each layer of convolutional or fully connected operators By running micro-benchmarks, the actual execution latency and power consumption of each operator are collected, and a latency prediction function and power consumption prediction function on the target platform are fitted using regression / neural network, as shown in the following formula.

[0155]

[0156]

[0157]

[0158] in, To determine the number of input channels for the l-th layer of the model to be deployed on the target platform, This refers to the number of output channels or the kernel size of layer l. is the dimension of the output feature map of layer l. To determine the true latency predictions for layer l of the model for deployment on the target platform, The energy consumption prediction value for layer l. This is the predicted peak memory value for layer l. function with , and As input, predict the latency of the model when executing a specific layer (such as a convolutional layer). Similarly , and As input, the model predicts the energy consumption of a specific layer. Similarly , and As input, the prediction model's maximum memory requirement during runtime.

[0159] Micro-benchmarks are used to collect runtime latency, power consumption, and GPU memory usage of convolution operators under different input channel numbers, kernel sizes, and output feature map dimensions. For example, when testing on the target automotive embedded platform, each layer of the model to be optimized is changed separately. , , And actually run it on the device to be deployed, measuring the actual inference latency at this layer on the target hardware platform. Since the test results are discrete, a continuous result is finally obtained through polynomial fitting. function.

[0160] Further, the scoring function is first calculated. For each channel... Define hardware perception score See the formula below.

[0161] +θ

[0162]

[0163]

[0164]

[0165]

[0166]

[0167]

[0168]

[0169] in, , , , These are the normalized results for weight norm, latency reduction, energy consumption reduction, and peak memory reduction, respectively. This is the original weight value of the channel. and These are the minimum and maximum weights across all channels, respectively. Similarly... , , and , , These are the minimum and maximum values ​​of latency, power consumption, and memory change across all channels.

[0170] These are hyperparameters, or preset weighting factors, used to balance the accuracy of the model and the hardware performance requirements, ensuring that the weighted average of the scores is meaningful. ,and Initialize by selecting a neutral starting point: .

[0171] Then, the pruning ratio of each layer is vectorized, which expands the scalar to a vector, as shown in the following formula.

[0172]

[0173] in, For layer The number of channels, Indicates the first The retention probability of each channel.

[0174] Then, a differentiable sampling mechanism is used, that is, during the training phase, the Gumbel Softmax Function (Gumbel-Softmax) is used to sample the channels while preserving their probabilities, thus obtaining a differentiable mask. See the formula below.

[0175]

[0176] The masked weight update is achieved using the following formula.

[0177]

[0178] in, This represents the original weight matrix of the l-th layer in the model. It is the pruning mask matrix designed for the l-th layer. It is the l-th layer after pruning masking The processed weight matrix.

[0179] Next, discretization pruning is performed; that is, after the search is completed, the discretization pruning is performed. Fixed, and based on global or local thresholds Discretize the data as shown in the following formula.

[0180]

[0181] The final sparse network structure is obtained.

[0182] Step 3, Hybrid precision quantization strategy.

[0183] First, perform layer / channel sensitivity analysis. For sensitive layers (such as the first layer, the last layer, or attention modules), maintain a high bit width (FP16 / INT8). For non-sensitive layers, use low bit quantization (INT4 / INT2).

[0184] Then, quantization is used to define a differentiable quantizable function, as shown in the following formula.

[0185]

[0186] in, For the first Number of quantization bits per layer.

[0187] Subsequent minor optimizations are possible. Treated as a continuous slack variable, and parameterized using softmax to ensure differentiability during the search. A quantization-aware loss is introduced during training, as shown in the following formula.

[0188]

[0189] in, , This is a weighting factor.

[0190] Finally, discretization is performed; that is, after the search is completed, the discretization process is performed. Rounded to the nearest valid bit width set And apply practical quantification.

[0191] Step 4: Fine-tuning and retraining.

[0192] For the pruned and quantized model, lightweight fine-tuning (LoRA or low-rank adapter) is used to restore accuracy, ensuring that the final model maintains performance while meeting hardware constraints.

[0193] Step 5, Hyperparameter adaptive update.

[0194] Initialize to neutral values ​​(e.g., [0.5, 0.3, 0.1, 0.1]), and use softmax constraints to make them differentiable during the search process. A hierarchical adaptive strategy can be adopted at different levels, with lower levels prioritizing latency and higher levels prioritizing accuracy.

[0195] As can be seen, this application achieves integrated optimization of structured pruning and hybrid precision quantization through differentiable search and hardware-aware modeling, including the following technical effects.

[0196] Technical effect 1: Significantly reduces model latency and energy consumption, and adapts to target hardware platforms.

[0197] Technical effect 2: Maintain or improve model accuracy by fine-tuning to compensate for pruning / quantization errors.

[0198] Technical effect 3: The pruning ratio and quantization bits can be differentiably optimized, improving search efficiency.

[0199] Technical effect 4: The pruning and quantification results can be directly deployed, with actual hardware acceleration effect.

[0200] Figure 2 This is another flowchart of a model training method according to one embodiment of this application, such as... Figure 2As shown, firstly, the "Setting of the Pruning and Quantization Joint Optimization Objective Function" defines the goal of seeking a balance between latency, energy consumption, storage, and accuracy. Then, the "Calibration of the Hardware-Aware Function" obtains performance metrics of specific operators on the target platform through micro-testing, providing hardware-level information support for subsequent decisions. Entering the "Parameter Differentiable Search" stage, the pruning ratio and quantization bit width are dynamically adjusted using differentiable architecture search technology to achieve coordinated optimization of model structure and accuracy. Next, the "Pruning and Quantizing the Model Based on Current Parameters" applies the currently searched parameters to prune and quantize the model, preparing for evaluation. In the "Evaluating the Model Based on the Objective Function" stage, the model performance is measured using the objective function to check whether it meets hardware resource constraints. If it does not meet the criteria, the evaluation result is fed back to the "Parameter Differentiable Search," adjusting the search direction until the optimal solution is found. This closed-loop feedback mechanism ensures that each search iteration is optimized based on the latest evaluation results, ultimately "obtaining the final optimized model," which significantly reduces hardware load while maintaining high accuracy, perfectly adapting to the stringent requirements of automotive embedded environments. The entire process demonstrates a high degree of integration between hardware awareness and model optimization, opening up a new optimization path for embedded applications.

[0201] Specifically, it includes the following steps.

[0202] S1, Establishment of joint optimization objectives.

[0203] S1.1, Set the latency budget on the target automotive embedded platform (such as Jetson Nano). Energy consumption budget Memory budget The latency budget can be determined based on the real-time requirements of the tasks the platform needs to complete, the power consumption budget can be determined based on the platform's rated power consumption, and the memory budget can be set based on the video memory size.

[0204] S1.2, Construct the joint optimization objective function:

[0205] in, These are the network weight parameters. The pruning ratio for each layer, Define the quantization bit width for each layer.

[0206] S1.3, employing the Differentiable Architecture Search (DARTS) method, while satisfying... Under constraints, joint optimization and .

[0207] S2, Target platform hardware-aware performance modeling.

[0208] S2.1 On the target in-vehicle embedded platform, adjust the number of input channels for each convolutional or fully connected operator in the model. Number of output channels or kernel size Output feature map dimension .

[0209] S2.2, Run the micro-benchmark program to actually measure the runtime latency of this layer. Energy consumption Peak memory usage .

[0210] S2.3, Perform multinomial regression or neural network fitting on the discrete sampled data to obtain a continuous prediction function:

[0211]

[0212] S2.4, Based on the above prediction results, define the hardware perception scoring function:

[0213]

[0214] in, These are the normalized values ​​of the channel's weight norm, latency reduction, energy reduction, and memory reduction, respectively. .

[0215] S3, Hardware-Aware Pruning Search.

[0216] S3.1, expand the hierarchical pruning ratio into a channel probability vector:

[0217]

[0218] S3.2, the retention probability of each channel is added to the hardware-aware score, and Gumbel-Softmax sampling is used to obtain the channel mask:

[0219]

[0220] S3.3, Multiply the channel mask by the weights to obtain the masked weight matrix:

[0221]

[0222] S3.4, During the search phase, the retention probability is dynamically updated. After the search is completed, the mask is... Discretize:

[0223]

[0224] This leads to the final sparse network structure.

[0225] S4, a mixed-precision quantization strategy.

[0226] S4.1 Perform sensitivity analysis on each layer, mark sensitive layers (such as the first layer, the last layer, and the attention module), and set their quantization bits to 8–16 bits; for non-sensitive layers, set their quantization bits to 2–8 bits.

[0227] S4.2, Define a differentiable function:

[0228]

[0229] S4.3, quantize bit width Relaxation is a continuous variable, differentiability is guaranteed by softmax, and a quantization-aware loss function is introduced during training:

[0230]

[0231] S4.4, After the search is complete, Discretize into sets The most recent valid number of bits in.

[0232] S5, model fine-tuning and retraining.

[0233] S5.1 After completing pruning and quantization, the model is fine-tuned.

[0234] S5.2 employs a lightweight adapter (such as LoRA or low-rank adaptation) to train for several rounds with a smaller learning rate in order to restore model accuracy.

[0235] S5.3 yields the final compression model that satisfies the hardware constraints.

[0236] S6, hyperparameter adaptive update.

[0237] S6.1, Initialize weight parameters It is a neutral value (e.g., [0.5, 0.3, 0.1, 0.1]).

[0238] S6.2 In the search process, the weight coefficients are constrained by softmax and differentiable optimization is performed in the training iteration.

[0239] S6.3 adopts a layered strategy, with different layers dynamically allocating weights: the bottom layer focuses on latency optimization, while the top layer focuses on accuracy preservation.

[0240] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0241] According to an embodiment of this application, an in-vehicle visual perception platform is provided. Figure 3 This is a structural schematic diagram of an in-vehicle visual perception platform according to one embodiment of this application, such as... Figure 3 As shown, the vehicle-mounted visual perception platform is equipped with a model, which is trained according to the model training method in any of the above embodiments.

[0242] For details, please refer to the description of the foregoing embodiments, which will not be repeated here.

[0243] According to an embodiment of this application, a model training device is provided. It should be noted that the device can be used to execute the above-described model training method.

[0244] Figure 4 This is a structural block diagram of a model training apparatus according to one embodiment of this application, such as... Figure 4 As shown, a model training device 400 is used as an example. This device includes: a construction module 401, used to construct an initial model and an objective function based on the hardware parameters of the target platform, wherein the hardware parameters represent the computing power, energy efficiency, and storage limitations of the target platform, and the objective function constrains the inference latency, energy consumption, and storage usage of the initial model; a determination module 402, used to determine the pruning ratio and a first bit width based on the initial weight parameters and the objective function of the initial model; a processing module 403, used to discretize the pruning ratio to obtain a pruning mask, and to discretize the first bit width to obtain a second bit width; an adjustment module 404, used to adjust the initial weight parameters based on the pruning mask and the second bit width to obtain target weight parameters; and a generation module 405, used to generate a target model based on the target weight parameters, wherein the target model is used to perform a deep learning task.

[0245] Embodiments of this application also provide a vehicle, including: a memory storing an executable program; and a processor for running the program, wherein the program executes the methods described in various embodiments of this application when it runs.

[0246] Embodiments of this application also provide a computer-readable storage medium including a stored executable program, wherein, when the executable program is running, it controls the device where the computer-readable storage medium is located to perform the methods of various embodiments of this application.

[0247] Embodiments of this application also provide a computer program product, including a computer program that, when executed by a processor, implements the methods of various embodiments of this application.

[0248] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0249] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0250] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0251] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0252] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0253] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A model training method, characterized in that, The method includes: An initial model and objective function are constructed based on the hardware parameters of the target platform, wherein the hardware parameters represent the computing power, energy efficiency and storage limitations of the target platform, and the objective function constrains the inference latency, energy consumption and storage usage of the initial model. Based on the initial weight parameters of the initial model and the objective function, the pruning ratio and the first bit width are determined; The pruning ratio is discretized to obtain a pruning mask, and the first bit width is discretized to obtain a second bit width. Based on the pruning mask and the second bit width, the initial weight parameters are adjusted to obtain the target weight parameters; A target model is generated based on the target weight parameters, wherein the target model is used to perform deep learning tasks.

2. The method according to claim 1, characterized in that, The process of constructing the initial model and objective function based on the hardware parameters of the target platform includes: The neural network parameters are determined based on the hardware parameters, wherein the neural network parameters include the number of input channels, the number of output channels, and the dimension of the output feature map; Micro-benchmark tests are performed on the target platform based on the neural network parameters to determine the performance data of the target platform, wherein the performance data includes runtime latency, power consumption, and peak memory usage; The performance data is subjected to multinomial regression or neural network fitting to obtain a prediction function, wherein the prediction function is used to predict the performance of the model on the target platform; The initial model and the objective function are constructed based on the prediction function.

3. The method according to claim 2, characterized in that, The prediction function includes a latency prediction function, an energy consumption prediction function, and a memory usage prediction function. The latency prediction function is used to predict the layer operation time, the energy consumption prediction function is used to predict the energy consumption of the layer operation, and the memory usage prediction function is used to predict the peak memory usage of the layer operation.

4. The method according to claim 1, characterized in that, The determination of the pruning ratio and the first bit width based on the initial weight parameters of the initial model and the objective function includes: The initial pruning ratio is obtained by initializing the pruning ratio based on the initial weight parameters and the objective function, and the initial bit width is obtained by initializing the bit width based on the initial weight parameters and the objective function. Based on the differentiable architecture search strategy and the target sampling strategy, the initial pruning ratio is iteratively updated to obtain the pruning ratio, and based on the differentiable architecture search strategy and the target sampling strategy, the initial bit width is iteratively updated to obtain the first bit width.

5. The method according to claim 4, characterized in that, The steps of initializing the pruning ratio based on the initial weight parameters and the objective function to obtain the initial pruning ratio, and initializing the bit width based on the initial weight parameters and the objective function to obtain the initial bit width, include: The hardware perception score is calculated based on the initial weight parameters and the objective function, wherein the hardware perception score includes hyperparameter values, which are used to represent the priority of different performance indicators; The pruning ratio is initialized based on the hyperparameter values ​​to obtain the initial pruning ratio, and the bit width is initialized based on the hyperparameter values ​​to obtain the initial bit width.

6. The method according to claim 5, characterized in that, The process of iteratively updating the initial pruning ratio based on the differentiable architecture search strategy and the target sampling strategy to obtain the pruning ratio, and iteratively updating the initial bit width based on the differentiable architecture search strategy and the target sampling strategy to obtain the first bit width, includes: Based on the differentiable architecture search strategy, the target sampling strategy, and the hyperparameter values, the initial pruning ratio is iteratively updated to obtain the pruning ratio. Additionally, based on the differentiable architecture search strategy, the target sampling strategy, and the hyperparameter values, the initial bit width is iteratively updated to obtain the first bit width. The hyperparameter values ​​are updated in each iteration according to the model state and the hardware parameters.

7. The method according to claim 1, characterized in that, The discretization of the pruning ratio to obtain the pruning mask includes: The pruning ratio is discretized based on a preset threshold to obtain the pruning mask, wherein the preset threshold is used to distinguish between channel retention and pruning.

8. The method according to any one of claims 1-7, characterized in that, The adjustment of the initial weight parameters based on the pruning mask and the second bit width to obtain the target weight parameters includes: Based on the preset fine-tuning algorithm, the pruning mask, and the second bit width, the initial weight parameters are fine-tuned to obtain the target weight parameters.

9. An in-vehicle visual perception platform, characterized in that, The vehicle-mounted visual perception platform is equipped with a model, which is trained using the model training method described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program is configured to execute the model training method described in any one of claims 1 to 8 when run on a computer or processor.

Images