A method for lightweight deployment of deep learning models on edge nodes in autonomous driving scenarios

Abstract

The present invention provides a method for lightweight deployment of deep learning models on edge nodes in autonomous driving scenarios, comprising the following steps: 1) confirming the deep learning model and data set to be used; 2) using a structured pruning method to reduce the size of the model during training and retaining the accuracy of the model as much as possible; 3) deploying the pruned model to the edge node; 4) retraining the model based on the localized data of the edge node combined with the original model. The method prunes the model trained based on cloud data, and downloads the pruned smaller model to the edge for retraining and fine-tuning.

Description

Technical Field

[0001] The present invention relates to the technical field of intersection of artificial intelligence and edge model deployment, and specifically to a method for lightweight deployment of deep learning models on edge nodes in autonomous driving scenarios. Background Art

[0002] As more and more deep learning models have made significant breakthroughs, DNN models have achieved excellent performance in computer vision tasks and are widely used in the field of autonomous driving. However, performing such DNN model computing tasks generally requires solving many challenges. For large-scale DNN models for traditional visual tasks, they are usually oriented to specific data sets and fully trained in the cloud. Such a training process often requires hundreds of hours of execution on a data center server cluster equipped with a large number of GPU computing cards and high-bandwidth network links. Due to the particularity of autonomous driving, real-time road condition videos collected need to be parsed immediately. Therefore, in the current autonomous driving application scenarios, the model needs to be deployed locally in the motor vehicle after being trained in the cloud. Through high-resolution video data and other sensor data, the autonomous driving model can perceive the current road, pedestrian and vehicle conditions in real time and perform inference calculations locally to complete autonomous driving related tasks, such as adjusting vehicle speed, changing lanes, etc. However, due to the complexity of autonomous driving tasks, special scenarios may be encountered during specific driving processes. The model performance can be improved by further collecting driving video data in special scenarios and performing a retraining process on these data. This is because there are more or less partial distribution differences between the data used for pre-training of the autonomous driving model and the real data in the user's road driving scenario. It is unrealistic to return all of this real-time driving data to the cloud central server. First, this means extremely high communication overhead. Taking 1080p data as an example, 5,000 cars driving for one day can generate 0.27TB of data. Second, for user privacy reasons, collecting driving data to the cloud may not be a good choice.

[0003] In recent years, edge computing solutions have been proposed. Edge computing deploys computing power closer to the data, reducing the data communication overhead required by traditional centralized cloud computing, while better protecting user privacy from being leaked. We hope to perform retraining on the vehicle based on the video data obtained during real-time driving to improve the performance of the autonomous driving model, and after collecting enough incremental models, return them to the cloud in a unified manner to achieve efficient distributed training.

[0004] However, there are many challenges in performing model retraining on resource-constrained vehicle computers. Existing model training stacks make full use of dedicated computing hardware to accelerate the training process, such as using high-performance GPU computing cards to process data in parallel. Usually, general vehicle computers do not have such hardware configurations. Vehicle computers are often equipped with arm-based CPUs to perform functions such as interaction with users. Secondly, segmenting models or data in autonomous driving scenarios often faces additional difficulties. Autonomous driving systems need to process data from multiple sensors, including but not limited to cameras, radars, lidars, and ultrasonic sensors. Each sensor provides different types of data, which may vary greatly in format, size, and time resolution. It is a challenge to integrate and synchronize these heterogeneous data for effective model training. For user privacy reasons, video data collected by a vehicle in real time is difficult to be processed on another vehicle. At the same time, there may not be a high-speed and stable bandwidth connection between vehicles to perform data exchange. Finally, the model training process is a task with a large amount of computation, and the data size can reach TB level. Even in cloud data centers, after obtaining additional data distribution, hundreds of epochs need to be executed to make the loss function converge. Each epoch may mean a complete traversal of the entire dataset, which is particularly time-consuming when the data scale is large. In cloud data centers, although computing resources are relatively sufficient, such large-scale data processing and model training still puts significant pressure on resources, resulting in a significant increase in computing costs. In addition, each epoch requires a large amount of data reading, processing, and model parameter update operations, which not only increases the computing time, but may also introduce additional network transmission delays, especially when data storage and computing resources are distributed in different geographical locations.

[0005] To this end, a method for lightweight deployment of deep learning models on edge nodes in autonomous driving scenarios is designed. Summary of the invention

[0006] The purpose of the present invention is to overcome the shortcomings of the prior art and to provide a method for lightweight deployment of deep learning models on edge nodes in autonomous driving scenarios. The method prunes the model trained based on cloud data and downloads the pruned smaller model to the edge for retraining and fine-tuning.

[0007] The present invention is implemented by the following technical solution: a method for lightweight deployment of deep learning models on edge nodes in an autonomous driving scenario, comprising the following steps:

[0008] 1) Confirm the deep learning model and dataset used;

[0009] 2) Use structured pruning methods during training to reduce the size of the model while retaining the accuracy of the model as much as possible;

[0010] 3) Deploy the pruned model to the edge node;

[0011] 4) Retrain the original model based on the localized data of the edge nodes.

[0012] Preferably, the pruning method in step 2) is specifically:

[0013] 1) Determine the type of deep learning model and data set, and clean and preprocess the data set;

[0014] 2) Confirm the maximum size of the model and adjust the algorithm's hyperparameters, including model learning rate, number of trainings, pruning rate, model weights, etc., and then prune the trained model;

[0015] 3) For the pruned model, evaluate whether the model scale needs to be adjusted. If necessary, adjust the hyperparameters and repeat step 2). If not, deploy the model.

[0016] Preferably, the local data of the edge node in step 4) is combined with the retraining of the original model as follows:

[0017] 1) Load the model obtained by the above method, and preprocess the pipeline and train the model;

[0018] 2) Extract local data on edge nodes and clean and preprocess the data;

[0019] 3) Use the new local data and the original learning model parameters to retrain the model through the Fit method;

[0020] 4) Compare the parameters of the retrained model with those of the original model to determine whether the retraining is effective;

[0021] 5) Fine-tune the retrained model and deploy it on the edge device for inference.

[0022] The beneficial effects of the present invention are as follows:

[0023] Compared with the prior art, the present invention prunes the model trained based on cloud data and downloads the pruned smaller model to the edge for retraining and fine-tuning. The positive effects achieved include:

[0024] (1) This paper proposes a distributed model retraining method under the edge computing architecture, which can be used in scenarios where massive data is generated in a decentralized manner. By deploying the model retraining process to the data source, we effectively reduce the communication overhead caused by the centralized data aggregation in the traditional retraining process.

[0025] (2) The present invention proposes a novel pruning algorithm, which uses pruning methods with corresponding characteristics according to different types of layers, so that the model size can be greatly reduced while ensuring that the accuracy does not decrease, thereby reducing the computational cost of retraining.

[0026] (3) The evaluation of the present invention on a real autonomous driving dataset shows that compared with directly reasoning with the trained model, our method can reduce the model size by 74.6% while increasing the model accuracy by 8.93%, which is well adapted to autonomous driving scenario problems. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 Schematic diagram of the architecture for lightweight deployment of deep learning models on edge nodes, using pruning and retraining methods.

[0028] Figure 2 Schematic diagram of the lightweight deployment method for deep learning models. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to more clearly understand the objectives, technical solutions and advantages of the present invention, the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0030] In the description of the present invention, it is necessary to understand that the orientation or position relationship indicated by the terms such as "up", "down", "left", "right", "inside", "outside", "horizontal" and "vertical" is based on the orientation or position relationship shown in the accompanying drawings, and is only for the convenience of describing the present invention, rather than indicating or implying that the device or original referred to must have a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0031] The present invention will be described in detail below with reference to the accompanying drawings: Figure 1-2 As shown, a method for lightweight deployment of deep learning models on edge nodes in an autonomous driving scenario includes the following steps:

[0032] 1) Confirm the deep learning model and dataset used;

[0033] 2) Use structured pruning methods during training to reduce the size of the model while retaining the accuracy of the model as much as possible;

[0034] 3) Deploy the pruned model to the edge node;

[0035] 4) Retrain the original model based on the localized data of the edge nodes.

[0036] The pruning method in step 2) is specifically as follows:

[0037] 1) Determine the type of deep learning model and data set, and clean and preprocess the data set;

[0038] 2) Confirm the maximum size of the model and adjust the algorithm's hyperparameters, including model learning rate, number of trainings, pruning rate, model weights, etc., and then prune the trained model;

[0039] 3) For the pruned model, evaluate whether the model scale needs to be adjusted. If necessary, adjust the hyperparameters and repeat step 2). If not, deploy the model.

[0040] The local data of the edge node in step 4) is combined with the retraining of the original model as follows:

[0041] 1) Load the model obtained by the above method, and preprocess the pipeline and train the model;

[0042] 2) Extract local data on edge nodes and clean and preprocess the data;

[0043] 3) Use the new local data and the original learning model parameters to retrain the model through the Fit method;

[0044] 4) Compare the parameters of the retrained model with those of the original model to determine whether the retraining is effective;

[0045] 5) Fine-tune the retrained model and deploy it on the edge device for inference.

[0046] Example 1

[0047] For ease of understanding, the method of the present invention is described in detail below with reference to formulas.

[0048] The pruning method of the present invention comprises:

[0049] 1) Determine the deep learning model The dataset is Divide the dataset into training sets Test Set and validation set And there is

[0050] 2) Confirm the model scale And ensure the scale of the model after training Adjust the learning rate lr, the number of training epochs, the pruning rate α, and the weights w of each layer of the model i , then start training and pruning.

[0051] 3) If the trained model satisfy Then the model Deploy to the edge node, otherwise adjust the hyperparameters and repeat step 2).

[0052] The local data of the edge node in the present invention is combined with the retraining of the original model, including:

[0053] 1) The model obtained by the above method Load and extract information such as the structure and weight parameters of each layer of the model.

[0054] 2) Process the local data of the edge node to obtain the local data set

[0055] 3) Confirm the hyperparameters for retraining: learning rate lr′, number of training epochs′, pruning rate α′, and weights w of each layer of the model i ′, etc., and retrain the model through the Fit method.

[0056] 4) If the difference in model weights is greater than the threshold ∈, that is, |w i -w′ i |≥∈, then retraining is effective, otherwise retraining parameters are readjusted for training.

[0057] 5) Retrain the completed model Redeploy to the edge node.

[0058] Anyone familiar with the technology may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. A method for lightweight deployment of deep learning models on edge nodes in autonomous driving scenarios, characterized in that: The following steps are involved: 1) Confirm the deep learning model and dataset used; 2) Use structured pruning methods during training to reduce the size of the model while retaining the accuracy of the model as much as possible; 3) Deploy the pruned model to the edge node; 4) Retrain the original model based on the localized data of the edge nodes.

2. The method for lightweight deployment of deep learning models on edge nodes in an autonomous driving scenario according to claim 1, characterized in that: The pruning method in step 2) is specifically as follows: 1) Determine the type of deep learning model and data set, and clean and preprocess the data set; 2) Confirm the maximum size of the model and adjust the algorithm's hyperparameters, including model learning rate, number of trainings, pruning rate, model weights, etc., and then prune the trained model; 3) For the pruned model, evaluate whether the model scale needs to be adjusted. If necessary, adjust the hyperparameters and repeat step 2). If not, deploy the model.

3. The method for lightweight deployment of deep learning models on edge nodes in an autonomous driving scenario according to claim 1, characterized in that: The local data of the edge node in step 4) is combined with the retraining of the original model as follows: 1) Load the model obtained by the above method, preprocess the pipeline and train the model; 2) Extract local data on edge nodes and clean and preprocess the data; 3) Retrain the model using the new local data and the original learning model parameters through the Fit method; 4) Compare the parameters of the retrained model with those of the original model to determine whether the retraining is effective; 5) Fine-tune the retrained model and deploy it on edge devices for inference.

Images