Automatic driving data automatic labeling system based on motorcade consensus mechanism

Abstract

The invention discloses an automatic driving data automatic labeling system based on a motorcade consensus mechanism, and relates to the technical field of automatic driving, and the specific scheme is as follows: respectively calculating corresponding labeling information and corresponding decision control information errors in a similar scene data set by using a loss function; and selecting the decision control information with the minimum error and labeling the corresponding original data in the similar scene data set to obtain a training data set. According to the scheme provided by the invention, the automatic labeling of the data is realized, the links of manually sorting and labeling the data afterwards are omitted, the efficiency and quality of automatic driving technology research and development are greatly improved, meanwhile, the cost is saved, and a training sample is provided for model training.

Description

Technical Field

[0001] The present invention relates to the fields of autonomous driving technology and artificial intelligence technology, and particularly to data collection and automatic data annotation, aiming to solve the problems of low processing efficiency of sample data and difficult reliable training of an autonomous driving model in autonomous driving technology, and specifically relates to an autonomous driving data automatic annotation system based on a fleet consensus mechanism. Background Art

[0002] "Automated Driving Classification for Motor Vehicles" (GB / T 40429-2021) classifies driving automation into levels 0 to 5, each level having corresponding technical requirements. As a necessary condition for the application and implementation of autonomous driving technology, data collection and data annotation have become an essential part of supporting autonomous driving technology.

[0003] Among them, data collection mainly includes two methods: manual and simulation. The manual method is to collect raw data such as space, vision, object shape, speed, etc. of the driving environment through a large number of diverse sensors installed on the collection vehicle, and then a large number of computer engineers organize, classify, cut, outline, describe, etc. these raw data to achieve data annotation, thereby obtaining the process of basic sample data. The simulation method is a process of automatically generating an environment perception data set with annotations based on a certain model algorithm or through the operation of professional engineers.

[0004] The manual method requires consuming a large amount of human resources and time costs, and it is difficult to guarantee the accuracy of data annotation; the simulation method largely depends on the technical strength of the simulation software provider and its own data accumulation, and also requires a large amount of human resources and time costs, and it is difficult for the simulation environment to cover or reflect the latest and timely actual physical operating environment conditions, thus it is difficult to achieve reliable training of the autonomous driving algorithm model.

[0005] The above content is only used to assist in understanding the technical solution of the present invention, and does not represent an admission that the above content is prior art. Summary of the Invention

[0006] The main purpose of the present invention is to collect and automatically annotate a large amount of driving data, provide accurate, reliable, and end-to-end training sample data for the autonomous driving algorithm model, and eliminate the high cost and low efficiency of relying on a large amount of manual annotation.

[0007] The present application proposes an autonomous driving data automatic annotation system based on a fleet consensus mechanism, including: a vehicle-end system and a cloud system connected through a communication module, where:

[0008] The vehicle-end system includes:

[0009] Data acquisition module: used to obtain vehicle status information and vehicle operating environment information and perform alignment processing to obtain raw data;

[0010] Pre-trained model: perform inference operations on the raw data to obtain corresponding annotation information;

[0011] Decision perception module: used to obtain the driving actions of the vehicle at the corresponding moment to obtain corresponding decision control information;

[0012] The vehicle-end system aligns the collected raw data, corresponding annotation information, and corresponding decision control information and sends them to the cloud system through the communication module;

[0013] The cloud system includes:

[0014] Central database: used to store the data sent by the vehicle-end system;

[0015] Automatic annotation module: classify the raw data, corresponding annotation information, and corresponding decision control information into a similar scenario data set based on the same scenario, similar scenarios, and / or similar actions, calculate the errors between the corresponding annotation information and corresponding decision control information in the similar scenario data set using a loss function, select the decision control information with the smallest error, and annotate the corresponding raw data in the similar scenario data set to obtain a training data set;

[0016] Pre-training system: train the model based on the training data set to obtain a trained pre-trained model.

[0017] Preferably, the alignment processing means that the collected raw data, corresponding annotation information, and corresponding decision control information are aligned in time.

[0018] Preferably, the similar scenario data set is a data set that obtains raw data, corresponding annotation information, and corresponding decision control information based on the same scenario, similar scenarios, and / or similar actions.

[0019] Preferably, the loss function includes mean squared error, cross entropy, or absolute error for different data types.

[0020] Preferably, the training data set is

[0021] Γ(D,T*)→Γ(D(n),T * )

[0022] where Γ(D(n),T * ) is the training data set; D(n) is the raw data at the nth moment in the similar data set; T * is the decision control information with the smallest error between the annotation information and the corresponding decision control information in the similar scenario data set; n = 1, 2, 3, ….

[0023] Preferably, the pre-trained model includes a recurrent neural network model, a convolutional neural network model, a multi-layer neuron model, or a deep neural network model based on a Transformer.

[0024] By adopting the solution of the present disclosure, a precise and efficient method for collecting and annotating vehicle driving behavior data is provided, eliminating the need for manual post-event data sorting and annotation, greatly improving the efficiency and quality of autonomous driving technology research and development, and also saving costs. Description of the Drawings

[0025] Figure 1 It is a schematic structural diagram of the vehicle-side system according to an embodiment of the present invention.

[0026] Figure 2 It is a flowchart of the data set according to an embodiment of the present invention.

[0027] Figure 3 It is a schematic structural diagram of the cloud system according to an embodiment of the present invention.

[0028] Figure 4 It is a flowchart of the model training method according to an embodiment of the present invention.

[0029] Figure 5 It is a schematic diagram of the reinforcement learning model according to an embodiment of the present invention.

[0030] Figure 6 It is a flowchart of the reinforcement learning method according to an embodiment of the present invention.

[0031] Figure 7 It is a framework of an autonomous driving data automatic annotation system based on a fleet consensus mechanism according to an embodiment of the present invention.

[0032] Reference Numerals and Names in the Drawings:

[0033] 10. Vehicle-side system; 101. Data collection module; 102. Pre-trained model; 103. Decision perception module;

[0034] 20. Fleet; 30. Similar scenario data set; 40. Similar data set; 50. Training data set;

[0035] 60. Cloud system; 601. Central database; 602. Automatic annotation module; 603. Pre-training system;

[0036] 701. Real-time environmental data; 702. Decision perception label; 703. Reward function; 704. Deep neural network.

[0037] The implementation, functional features, and advantages of the object of the present invention will be further described with reference to the embodiments and the accompanying drawings. Detailed Embodiments

[0038] To make the objectives, technical solutions, and advantages of the embodiments of the present disclosure clearer, the technical solutions of the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present disclosure. Apparently, the described embodiments are only a part rather than all of the embodiments of the present disclosure. All other embodiments obtained by those of ordinary skill in the art based on the described embodiments of the present disclosure without creative efforts shall fall within the scope of protection of the present disclosure.

[0039] The present invention provides a vehicle-end system, as Figure 1 shown. The vehicle-end system 10 includes a data acquisition module 101, a pre-trained model 102, and a decision perception module 103.

[0040] The vehicle-end system 10 refers to a system installed on a vehicle, mainly an automobile. It should be understood that it also includes other types of land vehicles, which are equipped with terminal devices such as sensors, control units, and communication modules, and can obtain vehicle status information and vehicle operating environment information in real time, and perceive and make judgments on the information through artificial intelligence algorithms, and control the vehicle to execute or not execute actions.

[0041] The vehicle-end system 10 can be a computer system installed on a vehicle or a unit with computer processing functions, which is responsible for transmitting the data collected by the sensors to the control unit, and through data processing, aligning the data in time, converting the format, and saving the data to obtain the original data at a certain moment, and annotating the original data, and performing inference operations and decision perception by model algorithms.

[0042] Multiple vehicles equipped with the vehicle-end system 10 form a vehicle fleet 20, and information interaction between vehicles and information interaction between vehicles and the cloud or traffic infrastructure can be realized through the communication module of the vehicle-end system 10. Among them, the communication module at least includes a V2X communication module, an Internet communication module, a 4G / 5G module, a WiFi module, etc.

[0043] The data acquisition module 101 obtains vehicle status information and vehicle operating environment information, including information such as space, vision, object shape, and speed, through sensors, at least including cameras, high-definition cameras, lidar, ultrasonic radars, and infrared sensors; for example, obtaining point cloud data through lidar, obtaining image data through cameras, and obtaining spatial distance data through ultrasonic radars.

[0044] The data acquisition module 101 collects data (including point cloud data, image data, and distance data) in chronological order, aligns the data in time, performs format conversion and data saving, and obtains the original data corresponding to a certain moment.

[0045] The pre-trained model 102 fuses the acquired information, sorts and semantically understands it in terms of time and space. For example, at a certain spatial position (relative to the vehicle) at a certain moment, an object or pedestrian with a certain motion state and form appears.

[0046] The pre-trained model 102 is a deep neural network model, including a recurrent neural network model, a convolutional neural network model, a multi-layer neuron model, and a deep neural network model based on a Transformer. It receives the raw data collected by the data acquisition module 101 and performs inference operations on the raw data through the pre-trained model 102 to obtain corresponding annotation information.

[0047] The decision-making and perception module 103 is connected to the control interface of the vehicle, obtains the driving actions at the current moment, makes real-time judgments and issues decision control information for the vehicle actions based on the current driving actions and the understanding of the perception data (annotation information), and converts the decision control information into actual control of the physical components of the vehicle, such as braking, steering, accelerating, decelerating, etc.; it can also obtain the driving actions of the vehicle at the current moment as decision control information and form a corresponding relationship between the annotation information and the decision control information.

[0048] The vehicle-end system 10 aligns the collected raw data, annotation information, and decision control information in time and packs or batches them into data packets for storage.

[0049] It can be known that for the autonomous driving system, it is crucial for the vehicle-end system of the present application that the pre-trained model 102 understands the perception data and makes correct decisions. The pre-trained model 102 requires a large amount of sample data for training. The sample data refers to the raw data information of the vehicle operating environment obtained through sensors and is annotated to give explanations of features, semantics, or final decisions, which is a data set containing the raw data and the corresponding annotation information.

[0050] In some embodiments, as Figure 2 shown, the data sets of the raw data, annotation information, and decision control information of the vehicle-end system 10 on a certain vehicle or different vehicles in the vehicle fleet 20 are classified as similar scenario data sets 30

[0051]

[0052] where g{} is the similar scenario data set; is a data set containing the raw data D, the annotation information PT, and the decision control information T; D(n) is the raw data at the nth moment; PT(n) is the corresponding annotation information of the raw data at the nth moment; T(n) is the decision control information at the nth moment; n = 1, 2, 3,....

[0053] The classification methods include the same scenario, such as the same intersection, the same road condition; and similar scenarios, such as classification according to similar intersections, similar road conditions; similar actions, such as actions like overtaking, lane changing, sudden braking, etc.; for the original data (D(1), D(2), …, D(n)) in the similar scenario dataset 30 obtained by the data acquisition module 101, the annotation information PT obtained through inference by the pre-trained model 102 is equal or the same, that is, the original data corresponding to the same annotation information PT can be classified into the similar dataset 40, so equation (1) is simplified to:

[0054]

[0055] Among them, g{} is the similar scenario dataset; is a data set containing the original data D, the annotation information PT, and the decision control information T; D(n) is the original data at the nth moment; PT is the annotation information corresponding to the similar dataset; T(n) is the decision control information at the nth moment; n = 1, 2, 3, ….

[0056] Use the loss function to calculate the error between the annotation information PT of the similar dataset 40 and the corresponding decision control information T respectively, and select the decision control information with the smallest error as a label T * .

[0057]

[0058] Among them, T * is the decision control information with the smallest error between the annotation information and the corresponding decision control information in the similar scenario dataset; represents the independent variable that obtains the minimum value within the range of T; δ(PT, T(n)) is the loss function of the error between the annotation information PT and the corresponding decision control information T in the similar dataset; n = 1, 2, 3, ….

[0059] The loss function of the error between the above-mentioned annotation information PT and the corresponding decision control information T includes mean square error, cross entropy, absolute error, etc. for different data types.

[0060] Re-label the label T * with the original data of the similar dataset to form a new training dataset 50.

[0061] Γ(D, T * ) → Γ(D(n), T * ) (4)

[0062] Among them, Γ(D(n), T * ) is the training dataset; D(n) is the original data at the nth moment in the similar data set.

[0063] The formed training data set 50 is used to train the pre-trained model, realizing automatic annotation of data, providing samples for model training, ensuring the stability of training, and improving the inference performance and accuracy of the model.

[0064] The present invention provides a cloud system for model training. As Figure 3 shown, the cloud system 60 includes a central database 601, an automatic annotation module 602, and a pre-training system 603.

[0065] The cloud system 60 can be a cloud server or a general server, where computer-readable instructions for model training are stored in the memory.

[0066] Central database 601: Used to store the similar scenario data set 30 and the iteratively updated data information. The similar scenario data set 30 is a data set of the original data, annotation information, and decision control information from the vehicle terminal systems 10 on different vehicles in the vehicle fleet 20.

[0067] The similar scenario data set is:

[0068]

[0069] where, g{} is the similar scenario data set; is a data set containing the original data D, annotation information PT, and decision control information T; D(n) is the original data at the nth moment; PT(n) is the corresponding annotation information of the original data at the nth moment; T(n) is the decision control information at the nth moment; n = 1, 2, 3,....

[0070] The similar scenario data set 30 is a data set obtained by acquiring the original data, corresponding annotation information, and corresponding decision control information based on the same scenario, similar scenarios, and / or similar actions.

[0071] Automatic annotation module 602: Using the loss function to calculate the error between the annotation information and the corresponding decision control information in the similar scenario data set 40 respectively, selecting the decision control information with the smallest error and annotating the original data in the similar scenario data set to obtain the training data set 50; based on the training data set 50, annotating the mapping relationship between the original data (input) and the decision control information (model inference result).

[0072] The training data set is:

[0073] Γ(D,T * )→Γ(D(n),T * ) (4)

[0074] where, Γ(D(n),T * ) is the training data set; D(n) is the original data at the nth moment in the similar data set; T* It is the decision control information with the smallest error between the annotation information and the corresponding decision control information in the similar scenario dataset; n = 1, 2, 3, ….

[0075] Loss functions include mean squared error, cross entropy, absolute error, etc. for different data types.

[0076] Pre-training system 603: Train the model in the pre-training system 603 based on the labeled training dataset to obtain a pre-trained model. The model can be a deep neural network model, including a recurrent neural network model, a convolutional neural network model, a multi-layer neuron model, and a deep neural network model based on Transformer.

[0077] Deploy the trained pre-trained model to the vehicle system 10 to form a continuously iterative vehicle networking platform, and realize the management of existing vehicles and fleets, as well as the collection and automatic annotation of a large amount of driving data, providing accurate, reliable, and end-to-end training sample data for the autonomous driving model, reliably training the model without manual annotation, and improving efficiency, accuracy, and reliability.

[0078] The present invention provides a model training method, as Figure 4 shown,

[0079] S101. Obtain vehicle state information and vehicle operating environment information and perform alignment processing to obtain raw data.

[0080] Obtaining vehicle state information and vehicle operating environment information includes information such as space, vision, object form, speed, etc.; for example, obtaining point cloud data through a lidar, obtaining image data through a camera, and obtaining spatial distance data through an ultrasonic radar.

[0081] Alignment processing means aligning the obtained information in time, that is, a certain moment corresponds to vehicle state information and vehicle operating environment information. Format conversion and storage are required for different format information.

[0082] S102. Based on the pre-trained model, label the raw data to obtain corresponding annotation information.

[0083] The pre-trained model is a deep neural network model, including a recurrent neural network model, a convolutional neural network model, a multi-layer neuron model, and a deep neural network model based on Transformer. It receives the raw data and performs inference operations on the raw data through the pre-trained model to obtain corresponding annotation information.

[0084] S103. Obtain the driving actions of the vehicle at the corresponding moment to obtain corresponding decision control information.

[0085] The driving action of the vehicle at a certain moment corresponds to the vehicle operating environment information at that moment and the annotation information corresponding to the original data at that moment.

[0086] S104. Classify the original data, the corresponding annotation information, and the corresponding decision control information into a similar scenario dataset.

[0087] The similar scenario dataset is:

[0088]

[0089] where g{} is the similar scenario dataset; is a data set containing the original data D, the annotation information PT, and the decision control information T; D(n) is the original data at the nth moment; PT(n) is the corresponding annotation information of the original data at the nth moment; T(n) is the corresponding decision control information at the nth moment; n = 1, 2, 3, ….

[0090] The similar scenario dataset is a data set that obtains the original data, the corresponding annotation information, and the corresponding decision control information based on the same scenario, similar scenarios, and / or similar actions.

[0091] S105. Use the loss function to calculate the errors between the corresponding annotation information and the corresponding decision control information in the similar scenario dataset respectively, select the decision control information with the smallest error, and label the corresponding original data in the similar scenario dataset to obtain the training dataset.

[0092]

[0093] where T * is the decision control information with the smallest error between the annotation information and the corresponding decision control information in the similar scenario dataset; represents the independent variable that obtains the minimum value within the range of T; δ(PT, T(n)) is the loss function of the error between the annotation information PT and the corresponding decision control information T in the similar dataset; n = 1, 2, 3, ….

[0094] The loss function includes mean square error, cross entropy, absolute error, etc. for different data types.

[0095] The training dataset is:

[0096] Γ(D, T * ) → Γ(D(n), T * ) (4)

[0097] where Γ(D(n), T * ) is the training dataset; D(n) is the original data at the nth moment in the similar data set; T *is the decision control information with the smallest error between the annotation information and the corresponding decision control information in the similar scenario dataset; n = 1, 2, 3, ….

[0098] S106. Train the pre-trained model based on the training dataset.

[0099] The pre-trained model can be a deep neural network model, including a recurrent neural network model, a convolutional neural network model, a multi-layer neuron model, and a deep neural network model based on a Transformer.

[0100] Deploy the trained pre-trained model into the original model to form a continuously iterative vehicle networking platform, and realize the management of existing vehicles and fleets, as well as the collection and automatic annotation of a large amount of driving data, providing accurate, reliable, and end-to-end training sample data for the autonomous driving model, training the model reliably, without manual annotation, and improving efficiency, accuracy, and reliability.

[0101] This method is a training method based on continuous data iteration of fleet collaboration. This method avoids time-consuming and laborious processing work such as manual data sorting, classification, cutting, outlining, and description, collects actual data to reflect the real physical operating environment, and ensures the reliability of model training.

[0102] Reinforcement learning is an effective method for training deep neural network models. This application proposes a reinforcement learning method, as Figure 5 and Figure 6 shown:

[0103] S201. Obtain real-time environmental data and corresponding decision perception labels to form a training dataset for training the deep neural network model.

[0104] Real-time environmental data 701: includes obtaining vehicle operating environment information, including space, vision, object shape, speed, etc. through sensors, at least including cameras, high-definition cameras, lidar, ultrasonic radars, and infrared sensors; for example, obtaining point cloud data through lidar, obtaining image data through cameras, and obtaining spatial distance data through ultrasonic radars.

[0105] Corresponding decision perception label 702: based on the real-time environmental data, the corresponding actions made by the vehicle, and this information obtains the corresponding driving actions through the vehicle connection interface, and annotates this information to obtain the corresponding decision perception label.

[0106] Use the real-time environmental data as the input and the corresponding decision perception label as the output to form a training dataset and train the deep neural network model to obtain a pre-trained model.

[0107] S202. Use the reward function to calculate the reward values of the real-time environment data and the corresponding decision perception labels in the training dataset, select the decision perception label with the largest reward value, and label the real-time environment data in the training dataset to obtain a new training dataset. Based on the new training dataset, train the training model.

[0108] The reward function 703 refers to the incentive for the optimal action of the decision perception label under the real-time environment data.

[0109] The reward value can reflect the safety of the vehicle during driving (including but not limited to the distance from the vehicle in front, whether it deviates from the lane, the prediction and avoidance of surrounding obstacles, etc.), comfort (including but not limited to vehicle speed, acceleration change, driving jerks, steering angular acceleration, etc.), and economy (including but not limited to fuel / electricity loss, vehicle component wear, etc.).

[0110] For example, in an emergency, a moving vehicle starts to brake until it stops, and the moving distance is a, and the distance between the vehicle and the obstacle is b. To avoid vehicle collision, it is necessary to satisfy a < b. Braking in an emergency will cause discomfort and even potential harm. How to brake within a limited braking distance, such as intermittent braking and controlling the length of the interval time, forms different strategies. Each strategy is scored and rewarded, and the higher the score, the better the comfort.

[0111] For a detailed introduction to the reward function, please refer to the relevant description in Patent CN 115107767A.

[0112] The decision perception label with the largest reward value is the optimal decision among the decisions made by different drivers in different vehicles in the fleet at different times under similar scenarios. Therefore, this decision perception label is the best decision obtained by comparing the vehicle's driving in time and space, and through continuous iteration of the reinforcement learning model, it is a dynamic optimization process and becomes the consensus mechanism or the autonomous awareness of the fleet.

[0113] The deep neural network 704 includes a recurrent neural network model, a convolutional neural network model, a multi-layer neuron model, and a deep neural network model based on the Transformer.

[0114] An information processor is configured to execute electronic instructions stored in a storage device. When the electronic instructions are executed, the information processor executes the above model training method and / or reinforcement learning method.

[0115] It should be noted that the vehicle-end system 10 and the cloud-end system 60 cooperate to execute the model training method.

[0116] This application also proposes an automatic annotation system for autonomous driving data based on the fleet consensus mechanism, such asFigure 7 As shown in the figure, it includes:

[0117] A vehicle-end system 10 and a cloud system 60 connected to the vehicle-end system 10 through a communication module; wherein:

[0118] The vehicle-end system 10 includes a data acquisition module 101, a pre-trained model 102, and a decision-making and perception module 103; the data acquisition module 101 is used to obtain vehicle state information and vehicle operating environment information and perform alignment processing to obtain raw data. The pre-trained model 102 is a deep neural network model, and the pre-trained model performs inference operations on the raw data to obtain corresponding annotation information. The decision-making and perception module 103 is used to obtain the driving actions of the vehicle at the corresponding moment to obtain corresponding decision-making and control information.

[0119] The vehicle-end system 10 sends the collected raw data, corresponding annotation information, and corresponding decision-making and control information to the cloud system 60 through the communication module; it can be known that the raw data, corresponding annotation information, and corresponding decision-making and control information are aligned in time, that is, a certain moment corresponds to vehicle state information and vehicle operating environment information. Format conversion and storage are required for different formats of information.

[0120] The communication module includes at least a V2X communication module, an Internet communication module, a 4G / 5G module, a WiFi module, etc. The sending forms of data include real-time transmission, batch sending, or offline copying.

[0121] The cloud system 60 includes a central database 601, an automatic annotation module 602, and a pre-training system 603; the central database 601 is used to store the data set sent by the vehicle-end system 10, including raw data, corresponding annotation information, and corresponding decision-making and control information.

[0122] The automatic annotation module 602 classifies the raw data, the corresponding annotation information, and the corresponding decision-making and control information into a similar scenario data set, calculates the errors of the corresponding annotation information and the corresponding decision-making and control information in the similar scenario data set using a loss function, selects the decision-making and control information with the smallest error, and annotates the corresponding raw data in the similar scenario data set to obtain a training data set.

[0123] Wherein: The similar scenario data set is:

[0124]

[0125] Wherein, g{} is the similar scenario data set; A data set containing original data D, annotation information PT, and decision control information T; D(n) is the original data at the nth moment; PT(n) is the corresponding annotation information of the original data at the nth moment; T(n) is the corresponding decision control information at the nth moment; n = 1, 2, 3, ….

[0126] The similar scenario data set is a data set that obtains original data, corresponding annotation information, and corresponding decision control information based on the same scenario, similar scenarios, and / or similar actions.

[0127]

[0128] Among them, T * is the decision control information with the smallest error between the annotation information and the corresponding decision control information in the similar scenario data set; represents the independent variable that obtains the minimum value within the range of T; δ(PT, T(n)) is the loss function of the error between the annotation information PT and the corresponding decision control information T in the similar data set; n = 1, 2, 3, ….

[0129] The loss function includes mean square error, cross entropy, absolute error, etc. for different data types.

[0130] The training data set is:

[0131] Γ(D, T * ) → Γ(D(n), T * ) (4)

[0132] Among them, Γ(D(n), T * ) is the training data set; D(n) is the original data at the nth moment in the similar data set; T * is the decision control information with the smallest error between the annotation information and the corresponding decision control information in the similar scenario data set; n = 1, 2, 3, ….

[0133] The pre-training system 603 trains the model based on the training data set to obtain the trained pre-training model.

[0134] The pre-training model can be a deep neural network model, including a recurrent neural network model, a convolutional neural network model, a multi-layer neuron model, and a deep neural network model based on Transformer.

[0135] Deploy the trained pre-training model into the original model to form a continuously iterative vehicle networking platform, and realize the management of existing vehicles and fleets, as well as the collection and automatic annotation of a large amount of driving data, providing accurate, reliable, and end-to-end training sample data for the autonomous driving model, reliably training the model, without manual annotation, improving efficiency, accuracy, and reliability.

[0136] The system avoids time-consuming and laborious processing work such as manual data sorting, classification, cutting, outlining, and description, collects actual data to reflect the real physical operating environment, and ensures the reliability of model training.

[0137] In the above embodiments, it can be implemented in computer hardware, software, or a combination thereof.

[0138] For example, to build an Internet of Vehicles platform for fleet management, a vehicle-end system 10 installed on several vehicles and a cloud-end system 60 connected to the vehicle-end system through the Internet. All the vehicles connected to the same cloud-end system 60 through the vehicle-end system 10 form a fleet 20. The vehicle-end system 10 is a set of computer programs running on multiple vehicles. The vehicle-end system 10 is connected to the cloud-end system 60 through the Internet to upload the data collected at the vehicle end, status perception, and decision signals. The cloud-end system 60 mainly includes a central database 601, an automatic data annotation module 602, and a pre-training system 603. The central database is used to store the data sent from the vehicle-end system and the data iteratively updated in the cloud-end system; the automatic annotation module is responsible for automatically tagging the original data. The pre-training system is responsible for training a deep neural network based on the labeled data samples to obtain a pre-trained model and upgrading and deploying it to the vehicle-end system.

Claims

1. Automatic labeling system for autonomous driving data based on fleet consensus mechanism, including: The vehicle-side system (10) and the cloud-side system (60) connected via the communication module are characterized in that: The vehicle-side system (10) comprises: Data acquisition module (101): used to acquire vehicle status information and vehicle operating environment information and perform alignment processing to obtain original data; Pre-training model (102): performs inference operations on the original data to obtain corresponding annotation information; A decision-making perception module (103): used to obtain the driving action of the vehicle at a corresponding moment and obtain corresponding decision-making control information; The vehicle-side system (10) aligns and processes the collected raw data, corresponding annotation information, and corresponding decision control information, and then sends the collected data to the cloud system (60) through a communication module; The cloud system (60) includes: Central database (601): used to store data sent by the vehicle-side system (10); Automatic labeling module (602): based on the original data, the corresponding labeling information and the corresponding decision control information, the original data, the corresponding labeling information and the corresponding decision control information are classified into similar scene data sets, and the error of the corresponding labeling information and the corresponding decision control information in the similar scene data sets is calculated by using a loss function, and the decision control information with the smallest error is selected and the corresponding original data in the similar scene data sets are labeled to obtain a training data set; Pre-training system (603): trains the model based on the training data set to obtain a trained pre-training model.

2. The automatic labeling system for autonomous driving data based on fleet consensus mechanism according to claim 1 is characterized in that: Alignment processing refers to the temporal alignment of the collected original data, the corresponding annotation information, and the corresponding decision control information.

3. The automatic labeling system for autonomous driving data based on fleet consensus mechanism according to claim 1 is characterized in that: The similar scene data set is a data set that obtains original data, corresponding annotation information and corresponding decision control information based on the same scene, similar scenes and / or similar actions.

4. The automatic labeling system for autonomous driving data based on a fleet consensus mechanism according to claim 1, characterized in that: The loss functions include mean square error, cross entropy or absolute error for different data types.

5. The automatic labeling system for autonomous driving data based on fleet consensus mechanism according to claim 1, characterized in that: The training data set is Γ(D,T * )→Γ(D(n),T * ) Among them, Γ(D(n),T * ) is the training data set; D(n) is the original data at the nth moment in the similar data set; T * The decision control information with the smallest error between the labeled information and the corresponding decision control information in the similar scene data set; n = 1, 2, 3, ….

6. The automatic labeling system for autonomous driving data based on fleet consensus mechanism according to claim 1, characterized in that: The pre-trained model includes a recurrent neural network model, a convolutional neural network model, a multi-layer neural network model or a transformer-based neural network model.

Images