A data training method for intelligent connected vehicles based on privacy data protection, electronic device and computer-readable storage medium

Abstract

The present invention discloses a data training method for intelligent networked vehicles based on privacy data protection. The method first labels the road test data collected historically, then performs model training, obtains the initial model, and deploys the low-level feature extraction layer of the initial model to the vehicle end; on the vehicle end, for the original data collected by the vehicle end, on the one hand, feature extraction is performed through the deployed low-level feature extraction layer to obtain a low-level feature data set and upload it to the cloud; on the other hand, the original data is anonymized and uploaded to the cloud for data labeling; on the cloud, the uploaded data set and the historical data set are used to train and update other feature extraction layers except the low-level feature extraction layer; and the updated model is pushed to the vehicle end for synchronous update. While ensuring data privacy transmission, the present invention solves the problem of algorithm performance degradation caused by anonymized data, and on this basis, constructs an algorithm closed loop to solve the problem of model iteration and update.

Description

Technical Field

[0001] The present invention relates to an improvement in data processing technology for intelligent network-connected vehicles, and in particular to an intelligent network-connected vehicle data training method based on privacy data protection, belonging to the technical field of data processing and training. Background Art

[0002] With the vigorous development of autonomous driving technology in China, major OEMs and technology suppliers are working hard to improve their competitiveness and have proposed various methods for data collection and algorithm training. For example: (1) Zhejiang Geely Holding Group Co., Ltd.'s CN202210057268 Data Collection Method, Device, Equipment and Storage Medium proposes the following technology: During the driving process of the vehicle, the vehicle-side data collected by the vehicle-side sensor and the road-side data collected by the road-side sensor are obtained, the vehicle-side data and the road-side data are synchronized in time and space, and the vehicle-side data and the road-side data after time and space synchronization are fused according to the high-precision map to obtain the target data, the scene classification is performed based on the target data, the scene data corresponding to multiple scenes are obtained, and the autonomous driving scene library is constructed according to the scene data. (2) Zhejiang Leapmotor Technology Co., Ltd.'s CN201910454082 L3 autonomous driving system road driving data collection, analysis and upload method proposes the following technologies: collecting vehicle-side driving data, including the collection and synchronization of driving data and the encoding and caching of driving data, performing online data analysis on the collected vehicle-side driving data, including the definition of the intermediate result output interface of the autonomous driving system, target matching consistency detection, positioning road sign semantic output, extreme vehicle operation detection and human-machine decision consistency detection, and then performing data communication to prepare for uploading of the vehicle-side driving data. Finally, the server receives and stores the vehicle-side driving data.

[0003] Therefore, more and more data is collected and used for algorithm training to improve the performance of autonomous driving, but there is a problem of user key information being leaked. In order to protect user key information from malicious use, relevant departments stipulate that when collecting video and image data outside the car, the car cannot directly provide unprocessed data to the outside of the car. If it needs to be provided to the outside of the car, the face and license plate information in the data must be anonymized inside the car. Therefore, how to reasonably and efficiently collect and use data for intelligent connected cars while protecting user privacy and security has become a key challenge.

[0004] The existing technology mainly anonymizes the data on the vehicle side and then uploads it to the cloud, where the anonymized data is used to train the model. A serious disadvantage of this type of technology is that some important information will be lost after the original data is anonymized, resulting in bias in the trained algorithm model when predicting non-anonymized data, which will produce large errors and affect the accuracy of the algorithm. When mass-produced vehicles are actually on the road, the algorithm performance will be reduced when using the original data for autonomous driving related functions.

[0005] In addition, the collection, training and deployment of autonomous driving data for mass-produced connected vehicles have not formed an algorithm closed loop, which reduces the speed of algorithm training and model iteration. Summary of the invention

[0006] In view of the above-mentioned deficiencies in the prior art, the purpose of the present invention is to provide a data training method for intelligent connected vehicles based on privacy data protection. While ensuring data privacy transmission, the present invention solves the problem of algorithm performance degradation caused by anonymized data, and on this basis constructs an algorithm closed loop to solve the problem of model iterative update.

[0007] The technical solution of the present invention is achieved in this way:

[0008] A data training method for intelligent connected vehicles based on privacy data protection includes the following steps:

[0009] 1) Obtaining the initial version of the model: In the cloud, the historically collected road test data is first annotated, and then the model is trained to obtain the initial version of the model. The low-level feature extraction layer of the initial version of the model (such as the first two convolutional layers) is deployed to the vehicle side;

[0010] 2) Extracting features from raw data on the vehicle side: On the vehicle side, extract features from the raw data collected in real time or historically on the vehicle side through the low-level feature extraction layer deployed in step 1) to obtain a low-level feature dataset of the raw data and upload it to the cloud.

[0011] 3) Desensitization of vehicle-side data: anonymize the key information in the original data on the vehicle side, upload the anonymized data to the cloud and annotate the data to obtain the annotated result data set;

[0012] 4) Preparation of cloud model update data: Match the data in step 3) of the labeled result dataset with the data in step 2) of the low-level feature dataset one by one to form a model update dataset;

[0013] 5) Model optimization: In the cloud, the model update dataset obtained in step 4) is used to train and update the feature extraction layers other than the low-level feature extraction layer in the initial model. The low-level feature extraction layer and the updated feature extraction layers are used as the optimized model, and the optimized model is pushed to the vehicle side for synchronous update.

[0014] Furthermore, in step 4), in the cloud, the road test data is subjected to feature extraction through the low-level feature extraction layer in the initial version of the model that is the same as the low-level feature extraction layer deployed to the vehicle side, to obtain a low-level feature data set of the road test data, and the union of the low-level feature data set of the road test data and the low-level feature data of the original data uploaded to the cloud is taken as the low-level feature data set of step 4).

[0015] Preferably, the model used in the model training in step 1) is a deep neural network.

[0016] Furthermore, the deep neural network includes but is not limited to convolutional neural networks, recurrent neural networks and their related variants, and the supported algorithms include but are not limited to target detection algorithms, lane line recognition algorithms, semantic segmentation algorithms, etc.

[0017] In step 3), the key information of the original data includes but is not limited to faces and license plates, and the anonymization processing includes mosaic, solid color filling, and blur processing.

[0018] The feature extraction method in step 2) of the present invention includes but is not limited to convolution, pooling, and slicing.

[0019] In step 1), the low-level feature extraction layers deployed to the vehicle side have multiple numbers of layers. Each time deployment is performed, low-level feature extraction layers of different numbers are deployed on the vehicle side at the same time. In step 5), multiple other feature extraction layers corresponding to the low-level feature extraction layers on the vehicle side are trained and updated in the cloud, thereby obtaining multiple optimized models, and synchronizing the model with the best performance to the vehicle side.

[0020] The model update dataset used for algorithm iterative update in step 4) includes but is not limited to the road test dataset, the original dataset collected by the vehicle, and other datasets obtained by data enhancement, including but not limited to data generated by flipping, rotating, and scaling low-level feature sets.

[0021] The present invention also provides an electronic device for data training of intelligent connected vehicles based on privacy data protection, comprising a memory configured to store executable instructions;

[0022] The processor is configured to execute executable instructions stored in the memory to implement the aforementioned intelligent connected vehicle data training method based on privacy data protection.

[0023] The present invention also provides a computer-readable storage medium on which computer program instructions are stored, and the computer program instructions execute the aforementioned intelligent connected vehicle data training method based on privacy data protection.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] 1. The present invention processes user key information by adding feature extraction operations and anonymization processing on the vehicle side, thereby protecting user privacy and security, and preventing key information from being leaked or uploaded, thereby meeting regulatory requirements.

[0026] 2. The present invention can retain the information of the original data to a large extent. Compared with the shortcoming of losing a large amount of useful information by relying on purely anonymized data for model training, the present invention can effectively improve the training effect of the algorithm.

[0027] 3. The present invention can realize the closed loop of data collection, labeling and training of intelligent connected vehicles, and can continuously improve the performance of autonomous driving algorithms after mass production without incurring large costs for vehicle-side road data collection and algorithm updates. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is a logical architecture diagram of the intelligent connected vehicle data training method based on privacy data protection of the present invention. DETAILED DESCRIPTION

[0029] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0030] The present invention proposes a connected car data collection, training and iteration method based on privacy data protection, such as Figure 1 As shown in the figure, it is divided into three parts: (1) cloud algorithm development before mass production; (2) pre-processing of mass production data collection; (3) algorithm iteration and update. Through these three parts, on the one hand, the user's privacy data security can be protected, and the information of original data and anonymized data can be fully utilized to improve the algorithm performance. On the other hand, it can form an algorithm closed loop and improve the ability of model iteration and update. The following is a detailed explanation of the three parts.

[0031] In the cloud algorithm development stage before mass production (i.e., the initial version of the model is obtained), the historically collected road test data is first annotated, such as pedestrians, vehicles, road signs, traffic lights and other information, and then the model is trained. Here, a deep neural network with feature extraction processing such as convolution, pooling, and slicing can be used, which can effectively extract image features and is often used in image classification, target recognition and other scenarios. Different models and depths are selected according to different target tasks. After repeated parameter adjustment and optimization, the initial version of the large mass production model is obtained, and the low-level feature extraction layer of the model is deployed to the vehicle side. The specific number of deployment layers is determined according to the processing power of the vehicle-side MCU and the data upload bandwidth. Fewer layers can retain more information about the input data, and more layers can extract higher-level features of the input data, and can also reduce the amount of data transmission, but more information will be lost, and the processing burden of the vehicle-side MCU will also be increased. The model training method of the present invention supports both incremental transfer learning and full algorithm training.

[0032] The mass production data collection and preprocessing stage, i.e. steps 2)-4), includes the collection and preprocessing of three parts of data. Inside the mass-produced vehicle, for the original picture and video data collected in real time or historically on the vehicle side, on the one hand, feature extraction is performed through the low-level convolutional layer deployed before mass production to obtain the low-level features of the original data and upload them to the cloud. Low-level features are some local features in the original data, which retain the relationship between the local and the whole, and are usually some straight line and curve features. Since feature extraction will lose some information, it is difficult to intuitively identify the target object with these feature data, which meets regulatory requirements and can be uploaded to the cloud. Similarly, in the cloud, the same feature extraction operation is performed on massive road test data to obtain a low-level feature data set of road test data. The union of the low-level feature data set of the road test data and the low-level feature data of the original data uploaded to the cloud is taken for the next model iteration. On the other hand, in mass-produced vehicles, by anonymizing key information in the original data, such as mosaicing faces and license plates, the anonymized data is uploaded to the cloud for data labeling. The anonymized pictures and videos do not affect the judgment of object categories and locations, so they can be accurately labeled without infringing user privacy. Data can be uploaded when the vehicle is in standby mode, which does not affect the performance of the vehicle-side algorithm and keeps the transmission stable.

[0033] In the algorithm iteration and update phase (i.e., model optimization), the cloud uses the low-level feature data set and the corresponding annotation result data set obtained in the previous phase for model training in the next phase, and can also be integrated with historical training data, which includes but is not limited to road test data sets and original data sets collected historically on the vehicle side, as well as other data sets that can be used for data enhancement method generation. Because the training data is obtained based on the low-level network, there is no need to train the entire network model at this stage, but only the high-level network part of the mass-produced large model needs to be trained and updated. The low-level feature extraction layer and other high-level feature extraction layers after training and updating are used as the optimized model. A certain push strategy, such as regular push or version update, is adopted to push the updated optimized model to the vehicle side for synchronous update, realizing the closed loop of autonomous driving data collection, training and deployment, and continuously improving the algorithm performance and driving experience when mass-produced vehicles are actually used.

[0034] While anonymizing the original data on the vehicle side, the present invention adds a feature extraction operation to obtain the low-level feature data of the original data, and uses these two parts of data to solve the problem of algorithm performance degradation caused by anonymized data, and on this basis constructs an algorithm closed loop to solve the problem of iterative model updates. This part of the data after feature extraction is no longer the original data, and uploading it to the cloud will not leak privacy information; the original data will not leak privacy information after being anonymized and then uploaded; anonymization processing and labeling will not affect the accuracy of the labeling. Therefore, the present invention proposes a closed-loop method for autonomous driving algorithms for data collection, labeling, and training of connected vehicles. The present invention also proposes an in-vehicle data processing method that can be applied to privacy protection and retains the original data information to a high degree.

[0035] Finally, it should be noted that the above examples of the present invention are merely examples for illustrating the present invention, and are not intended to limit the embodiments of the present invention. Although the applicant has described the present invention in detail with reference to the preferred embodiments, for those of ordinary skill in the art, other different forms of changes and modifications can be made based on the above description. It is impossible to list all the embodiments here. Any obvious changes or modifications derived from the technical solution of the present invention are still within the scope of protection of the present invention.

Claims

1. A data training method for intelligent connected vehicles based on privacy data protection, characterized in that: The following steps are included: 1) Obtaining the initial version of the model: In the cloud, the historically collected road test data is first annotated, and then the model is trained to obtain the initial version of the model. The low-level feature extraction layer of the initial version of the model is deployed to the vehicle side; 2) Extracting features from raw data on the vehicle side: On the vehicle side, extract features from the raw data collected in real time or historically on the vehicle side through the low-level feature extraction layer deployed in step 1) to obtain a low-level feature dataset of the raw data and upload it to the cloud. 3) Desensitization of vehicle-side data: anonymize the key information in the original data on the vehicle side, upload the anonymized data to the cloud and annotate the data to obtain the annotated result data set; 4) Preparation of cloud model update data: Match the data in step 3) of the labeled result dataset with the data in step 2) of the low-level feature dataset one by one to form a model update dataset; 5) Model optimization: In the cloud, the model update data set obtained in step 4) is used to train and update the feature extraction layers other than the low-level feature extraction layer in the initial model. The low-level feature extraction layer and the updated feature extraction layers are used as the optimized model, and the optimized model is pushed to the vehicle end for synchronous update. In step 4), in the cloud, the road test data is subjected to feature extraction by the same low-level feature extraction layer in the initial version of the model as the low-level feature extraction layer deployed on the vehicle side, to obtain a low-level feature data set of the road test data, and the union of the low-level feature data set of the road test data and the low-level feature data of the original data uploaded to the cloud is taken as the low-level feature data set of step 4); In step 1), the low-level feature extraction layers deployed to the vehicle side have multiple numbers of layers. Each time deployment is performed, low-level feature extraction layers of different numbers are deployed on the vehicle side at the same time. In step 5), multiple other feature extraction layers corresponding to the low-level feature extraction layers on the vehicle side are trained and updated in the cloud, thereby obtaining multiple optimization models, and the optimization model with the best performance is synchronized to the vehicle side as the optimized model.

2. The method for training data of intelligent connected vehicles based on privacy data protection according to claim 1, characterized in that: The model used in the model training of step 1) is a deep neural network; The deep neural network includes convolutional neural network, recurrent neural network and their related variants, and the supported algorithms include target detection algorithm, lane line recognition algorithm and semantic segmentation algorithm.

3. The method for training data of intelligent connected vehicles based on privacy data protection according to claim 1, characterized in that: In step 3), the key information of the original data includes faces and license plates, and the anonymization processing includes mosaic, solid color filling, and blur processing.

4. The method for training data of intelligent connected vehicles based on privacy data protection according to claim 1, characterized in that: The feature extraction methods used in step 2) include convolution, pooling, and slicing.

5. The method for training data of intelligent connected vehicles based on privacy data protection according to claim 1, characterized in that: Step 2) and step 3) upload the data to the cloud when the vehicle is in standby mode.

6. The method for training data of intelligent connected vehicles based on privacy data protection according to claim 1, characterized in that: The model update dataset used for algorithm iterative update in step 4) includes the road test dataset, the original dataset collected by the vehicle, and other datasets obtained by data enhancement, including data generated by flipping, rotating, and scaling low-level feature sets.

7. An electronic device for training data of intelligent connected vehicles based on privacy data protection, characterized in that: including a memory configured to store executable instructions; A processor is configured to execute executable instructions stored in a memory to implement a data training method for intelligent connected vehicles based on privacy data protection as described in any one of claims 1 to 6.

8. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: The computer program instructions execute a method for training data of intelligent connected vehicles based on privacy data protection as described in any one of claims 1 to 6.

Images