Automatic automobile driving method and device

[0034] The present invention provides a method for auto-driving a car. The method uses the collected environmental perception information to establish a driving environment risk field, and trains a deep learning model according to the driving environment risk field and driver operation, which can realize the automatic driving of the vehicle and reduce the automatic driving of the vehicle. The difficulty of training the driving model (referred to as the model).

[0035] figure 1 It is a schematic flowchart of an embodiment of the method for automatic driving of an automobile of the present invention. Such as figure 1 As shown, the method includes the following steps:

[0036] Step S102: Establish a vehicle automatic driving database according to the collected environmental perception information and driver operation information, and divide the vehicle automatic driving database into a training set and a test set to form a sample. The training set is used to train the model and is used in the model training phase; the test set is used to verify the usability of the model and is used in the model testing phase.

[0037] Among them, the environmental perception information is environmental data collected by at least one sensor. For example, the images collected by the vehicle-mounted camera, the point cloud information of the lidar and the target information of the millimeter-wave radar, but not limited to the examples.

[0038] Among them, the driver's operation information includes vehicle steering angle, vehicle acceleration/deceleration and other information. In order to obtain rich and diverse driving data, the driving data of different drivers can be selected. For example, the database sampling frequency used is 10 Hz, two hours of driving data of different drivers are selected as the training set, a total of 72,000 frames, and half an hour of driving data of different drivers is selected as the test set, a total of 18,000 frames. By learning the driving behavior of different drivers, it is possible to realize the anthropomorphic automatic driving of the vehicle.

[0039] Step S104: Establish a driving environment risk field for training according to the environmental perception information in the training set, and train the deep learning model according to the driving environment risk field for training and driver operation information in the training set.

[0040] In one embodiment, training the deep learning model includes: inputting the driving environment risk field for training and the driver operation information in the training set into the deep learning model, and outputting predicted vehicle control variables, such as vehicle steering angle, vehicle plus/ Deceleration, etc., determine the loss information of driver operation information based on predicted vehicle control variables and expected vehicle control variables (determine the desired vehicle control variables based on driver operation information), and modify the deep learning model based on the loss information of driver operation information The parameters of the vehicle control variables. After a certain number of iterations, a deep learning model that meets the requirements is obtained. Among them, the deep learning model may be, for example, a deep convolutional neural network model.

[0041] The driving environment risk field can fully describe the driving environment, which is conducive to the realization of automatic driving in different road environments.

[0042] Step S106: Establish a driving environment risk field for the test according to the environmental perception information in the test set, input the driving environment risk field for the test into the deep learning model, output the predicted vehicle control variables, and compare the predicted vehicle control variables with the test set Driver operation information tests the deep learning model.

[0043] An exemplary test method is described as follows. If the difference between the predicted vehicle control variables and the driver operation information in the test set is less than the preset value, that is, the consistency between the two is good, the deep learning model can be determined Available. Among them, vehicle control variables include, for example, vehicle steering angle, vehicle acceleration/deceleration, and so on.

[0044] However, those skilled in the art can understand that the above test method is not the only one. For example, use the predicted vehicle control variables to control the vehicle, observe whether the vehicle can drive normally, and if it can drive normally, determine that the deep learning model is available.

[0045] In the foregoing embodiment, the present invention establishes a risk model of the vehicle driving environment based on the field theory, so that the driving environment risk field is used to comprehensively reflect the driving environment of the vehicle, which is beneficial to realize automatic driving in different road environments. And according to the driving environment risk field and the driver's operation, the vehicle automatic driving model is trained to learn the experience of excellent human drivers and realize anthropomorphic automatic driving. In addition, compared with directly using environmental perception information to train the vehicle automatic driving model, the training difficulty of the vehicle automatic driving model is reduced.

[0046] The invention also provides a method for training the deep learning model. See figure 2 The flow chart for training the deep learning model is shown. For the data in the training set, the training process is as follows:

[0047] Step S202: Recognizing environment perception information collected by at least one sensor in the training set, and identifying driving environment information such as stationary objects, moving objects, and roads.

[0048] The following uses cameras, lidars, millimeter wave radars, etc. as examples to describe the recognition process.

[0049] According to the images collected by the camera, targets such as lane lines and vehicles are identified. Among them, image processing methods can be used to recognize lane lines in the image. This method achieves accurate identification and stable tracking of lane lines through the steps of image adaptive threshold segmentation, lane marking feature point extraction, feature point clustering and fitting, lane line matching and tracking. Among them, machine learning methods can be used to identify vehicle targets in images. This method uses HOG (Histogram of Oriented Gradient) features and AdaBoost (an iterative algorithm) cascade classifier to train the vehicle detection model, and then uses the vehicle detection model to achieve accurate detection of vehicle targets. Those skilled in the art can understand that for objects such as pedestrians, cyclists, roads, road signs, etc., reference can be made to the aforementioned identification methods of lane lines and vehicle objects, which will not be repeated here.

[0050] In addition, lidar can obtain point cloud information (ie, spatial coordinate information) of obstacles and road surfaces on the road. Millimeter-wave radar can obtain information such as the position and speed of obstacles (such as vehicles, fences and other targets).

[0051] In step S204, optionally, in the case where multiple sensors collect environmental perception information, coordinate conversion and/or data fusion may also be performed.

[0052] Among them, coordinate conversion refers to the conversion of the coordinate systems of multiple sensors to form a unified coordinate system, which makes subsequent data fusion easier. A coordinate conversion method can be, for example, to convert the image coordinate system to camera coordinates, and then to convert the camera coordinates and the coordinate system of other sensors to a unified vehicle coordinate system (for example, the coordinate system fixed to the own vehicle, and the coordinate origin is at Vehicle centroid) to realize the coordinate conversion of different sensor perception information.

[0053] Among them, due to the different attributes of the sensing information of different sensors, such as low lateral resolution of millimeter wave radar and poor ranging accuracy of vision sensors, the present invention uses the Mahalanobis distance of the target to correlate the same target observed by different sensors, and further in order to fuse different sensor observations. The value of the same target observed by different sensors is weighted and averaged according to the probability of occurrence, as the probability of occurrence of the same target, so as to realize the fusion of multi-sensor information and the effective estimation of the true state of the observation value. Among them, for example, a joint probability data association (JPDA, Joint Probability Data Association) method may be used to perform a weighted average of the same target observed by different sensors according to the probability of occurrence.

[0054] Through the above-mentioned coordinate conversion or data fusion, road environment information can be more accurately identified.

[0055] Step S206: Establish a driving environment risk field for training according to the environmental perception information in the training set.

[0056] The invention provides a method for establishing a risk field that can comprehensively reflect the risk degree of the vehicle driving environment. That is, the driving environment risk field is established according to the potential energy field information formed by stationary objects (such as parked vehicles, etc.), the kinetic energy field information formed by moving objects (such as moving vehicles and pedestrians), and the behavior field information formed by the driver. The formula is expressed as follows :

[0057] Es=Er+Ev+Ed (1)

[0058] Among them, Es represents the driving environment risk field, Er represents the potential energy field information formed by stationary objects, Ev represents the kinetic energy field information formed by moving objects, and Ed represents the behavior field information formed by the driver.

[0059] For the driving environment risk field used for training, the potential energy field information formed by stationary objects and the kinetic energy field information formed by moving objects are determined according to the environmental perception information in the training set, and the behavior field information formed by the driver is determined according to the driver operation information in the training set. Specifically, the potential energy field characterizes the physical field of the impact of stationary objects on the road on the safety of driving. The magnitude and direction of the potential energy field are mainly determined by the attributes of the stationary objects and road conditions. The kinetic energy field is a physical field that characterizes the impact of moving objects on the road on the safety of driving. The size and direction of the kinetic energy field are mainly determined by the properties of the moving objects, the state of motion and the road conditions. The behavior field is a physical field that characterizes the influence of the driver's behavior characteristics on driving safety. The strength of the behavior field is mainly determined by the driver's behavior characteristics. Under the same conditions, aggressive drivers often cause greater driving risks than conservative drivers, and their behavior field strength is greater; drivers with low driving skills are usually stronger than drivers with high driving skills.

[0060] image 3 Shows a schematic diagram of a driving environment risk field in a typical road environment. To facilitate the training process of deep learning, the risk field can be discretized and projected onto a two-dimensional image. Among them, the abscissa of the risk field image represents the lateral direction of the vehicle, the ordinate represents the longitudinal direction of the vehicle, and the image pixel value represents the degree of risk (for example, it can be quantified from 0 to 255). In this embodiment, for example, a range of 20 meters to the left and right, 100 meters to the front, and 50 meters to the rear of the vehicle can be considered. Each pixel represents a length of 0.5 meters, so the generated risk field grayscale image size is 300x80.

[0061] Step S208: Input the driving environment risk field for training and the driver's operation information (that is, supervision information) in the training set into the deep learning model, and output the predicted vehicle control variables.

[0062] Among them, the driver's operation information includes vehicle steering angle, vehicle acceleration/deceleration and other information. In order to obtain rich and diverse driving data, the driving data of different drivers can be selected.

[0063] Wherein, the deep learning model may be, for example, a deep convolutional neural network model, which includes five layers of convolutional layers and two fully connected layers, and the last layer outputs a two-dimensional vehicle control value.

[0064] Step S210: Determine the loss information of the driver's operation information according to the predicted vehicle control variables and the expected vehicle control variables (determined according to the driver's operation information), for example, use the L2 loss function to modify the deep learning model according to the loss information of the driver's operation information The parameters of the vehicle control variables.

[0065] After a certain number of iterations (for example, 100,000 times), a deep learning model that meets the requirements can be obtained, thereby completing the training process of the deep learning model.

[0066] In the foregoing embodiment, a risk assessment model of the vehicle driving environment is established based on the field theory, and multiple sensor information inputs are integrated to establish a comprehensive driving environment description system, which is beneficial to realize automatic driving in different road environments. Combining the driving environment of the vehicle and the driver's corresponding vehicle operation output, learning the auto-driving model of the vehicle based on the deep learning method can realize the auto-driving of the vehicle. By learning the driving behavior of different drivers, the anthropomorphic automatic driving of the vehicle can be realized.

[0067] The invention also provides a method for testing the deep learning model. See Figure 4 The flow chart for testing the deep learning model is shown. For the data in the test set, the testing process is as follows:

[0068] Step S402: Recognize the environment perception information collected by at least one sensor in the test set, and recognize driving environment information such as stationary objects, moving objects, and roads.

[0069] Among them, the method for recognizing environmental perception information in the test set can refer to the method for recognizing environmental perception information in the training set (ie, refer to step S202), which will not be repeated here.

[0070] In step S404, optionally, when multiple sensors collect environmental perception information, coordinate conversion and/or data fusion may also be performed.

[0071] Among them, the coordinate conversion and/or data fusion method of the environment perception information in the test set can refer to the coordinate conversion and/or data fusion method of the environment perception information in the training set (ie, refer to step S204), which will not be repeated here.

[0072] Step S406: Establish a driving environment risk field for the test according to the environmental perception information in the test set.

[0073] Wherein, the method for establishing the driving environment risk field for testing can refer to the method for establishing the driving environment risk field for training (ie, refer to step S206), which will not be repeated here.

[0074] Step S408: Input the driving environment risk field used for testing into the trained deep learning model, that is, use the trained deep learning model to process the input driving environment risk field, and output the predicted vehicle control variables.

[0075] For example, input the generated grayscale image of the risk field with a size of 300x80 into the trained deep convolutional neural network model, and obtain two-dimensional vehicle control variables through regression, such as vehicle steering angle, acceleration/deceleration and other information.

[0076] In step S410, the deep learning model is tested by comparing the predicted vehicle control variables with the driver operation information in the test set.

[0077] An exemplary test method is described as follows. If the difference between the predicted vehicle control variables and the driver operation information in the test set is less than the preset value, that is, the consistency between the two is good, the deep learning model can be determined Available. Among them, vehicle control variables include, for example, vehicle steering angle, vehicle acceleration/deceleration, and so on.

[0078] If the deep learning model is available, you can use PID (proportional integral derivative) control to achieve effective control of the vehicle based on the vehicle control variables output by the deep learning model (such as vehicle steering angle, acceleration/deceleration, etc.).

[0079] The invention also provides an automatic driving device for an automobile, with reference to Figure 5 , The device includes:

[0080] The sample forming module 502 is used to establish a vehicle automatic driving database according to the collected environmental perception information and driver operation information, and divide the vehicle automatic driving database into a training set and a test set;

[0081] The model training module 504 is configured to establish a driving environment risk field for training according to the environmental perception information in the training set, and train the deep learning model according to the driving environment risk field for training and driver operation information in the training set;

[0082] The model testing module 506 is used to establish a driving environment risk field for testing according to the environmental perception information in the test set, input the driving environment risk field for testing into the deep learning model, output predicted vehicle control variables, and compare the predicted vehicle control variables Test the deep learning model with the driver operation information in the test set.

[0083] reference Image 6 In the case that environmental perception information is collected by multiple sensors, the sample forming module 502 includes: a data processing unit 5022 and a sample forming unit 5024;

[0084] The data processing unit 5022 is used to transform the coordinate systems of multiple sensors to form a unified coordinate system; and/or use the Mahalanobis distance of the target to correlate the same target observed by different sensors, and press the same target observed by different sensors The probability of occurrence is weighted and averaged as the probability of occurrence of the same target.

[0085] The sample forming unit 5024 is used to establish a vehicle automatic driving database according to the collected environmental perception information and driver operation information, and divide the vehicle automatic driving database into a training set and a test set.

[0086] The model training module 504 includes a first risk field establishing unit 5042, which is used to establish a driving environment risk field for training according to the potential energy field information formed by stationary objects, the kinetic energy field information formed by moving objects, and the behavior field information formed by the driver; Among them, the potential energy field information formed by stationary objects and the kinetic energy field information formed by moving objects are determined according to the environmental perception information in the training set, and the behavior field information formed by the driver is determined according to the driver operation information in the training set.

[0087] The model training module 504 includes a model training unit 5044, which is used to input the driving environment risk field for training and the driver operation information in the training set into the deep learning model, and output the predicted vehicle control variables and loss information of the driver operation information; According to the loss information of the driver's operation information, the parameters of the vehicle control variables in the deep learning model are corrected.

[0088] Wherein, the model testing module 506 includes a second risk field establishing unit 5062, which is used to establish a driving environment risk field for testing according to the potential energy field information formed by stationary objects, the kinetic energy field information formed by moving objects, and the behavior field information formed by the driver; Among them, the potential energy field information formed by stationary objects and the kinetic energy field information formed by moving objects are determined according to the environmental perception information in the test set, and the behavior field information formed by the driver is determined according to the driver operation information in the test set.

[0089] Among them, the potential energy field information formed by a stationary object is determined according to the attributes of the stationary object and road conditions; the kinetic energy field information formed by a moving object is determined according to the attributes, movement state and road conditions of the moving object.

[0090] Among them, the model testing module 506 includes a model testing unit 5064, which is used to input the driving environment risk field for testing into the deep learning model and output the predicted vehicle control variables. By comparing the predicted vehicle control variables with the driver operation information in the test set Deep learning models are tested.

[0091] The present invention establishes a risk model of the vehicle driving environment based on the field theory, so that the driving environment risk field is used to comprehensively reflect the driving environment of the vehicle, which is beneficial to realize automatic driving under different road environments. And according to the driving environment risk field and the driver's operation, the vehicle automatic driving model is trained to learn the experience of excellent human drivers and realize anthropomorphic automatic driving. In addition, compared with directly using environmental perception information to train the vehicle automatic driving model, the training difficulty of the vehicle automatic driving model is reduced.

[0092] Finally, it should be pointed out that the above embodiments are only used to illustrate the technical solutions of the present invention, but not to limit them. A person of ordinary skill in the art should understand that the technical solutions described in the foregoing embodiments can be modified, or some of the technical features can be equivalently replaced; these modifications or replacements do not depart from the essence of the corresponding technical solutions of the present invention. The spirit and scope of the technical solutions of the embodiments.