An automatic parking method based on multi-source information perception and end-to-end deep learning

Abstract

The application belongs to the technical field of automatic parking and discloses an automatic parking method based on multi-source information sensing and end-to-end deep learning; four-way fisheye image data, ultrasonic radar data, steering wheel rotation angle and vehicle speed data in a parking process are sampled in real time to construct an initial data set; four-way fisheye image data in the initial data set is preprocessed into surround view image information data to construct training samples; a CNN-LSTM neural network is built and optimized, CNN is used for processing of the surround view image information data, and LSTM is used for processing of ultrasonic obstacle distance information data and current driver driving information data; the training samples are input into the CNN-LSTM neural network for training to obtain a trained end-to-end automatic parking model; real vehicle control is performed by using the automatic parking model to realize end-to-end automatic parking. The application solves the problems of low planning accuracy and slow response speed of the existing automatic parking method.

Description

Technical Field

[0001] This invention relates to the field of automatic parking technology, and in particular to an automatic parking method based on multi-source information perception and end-to-end deep learning. Background Technology

[0002] With societal progress and the continuous improvement of living standards in my country, cars have become an indispensable means of transportation. However, the increasing number of vehicles has led to urban congestion, a major problem that causes significant inconvenience to our travel. Automatic parking cars, also known as driverless parking cars, computer-controlled parking cars, or wheeled mobile robots, are intelligent vehicles that achieve driverless parking through computer systems.

[0003] The parking motion strategy based on path planning and path tracking first plans the parking path by combining the vehicle's kinematic constraints, and then uses a control algorithm for path tracking. (First, the parking space and vehicle attitude are estimated using sensors, and then the optimal parking path is planned). Traditional planning methods (e.g., circular arc method) can meet the requirements, but the constraints will increase accordingly, the solution process will become more complex, and the planning accuracy and response speed will be reduced. Summary of the Invention

[0004] To address the aforementioned problems and further improve the parking accuracy and response speed of automatic parking, this invention proposes an automatic parking method based on multi-source information perception and end-to-end deep learning to achieve end-to-end automatic parking.

[0005] To achieve the above objectives, the specific technical solution of the present invention is as follows: an automatic parking method based on multi-source information perception and end-to-end deep learning, comprising the following steps:

[0006] 1) The parking process is sampled at a sampling frequency f to construct an initial dataset D; the initial dataset is denoted as D = {d1, d2, ..., dn}. i ……},d i Let d be the sampled data of the i-th time. i ={Pf i Pb i ,Pl i ,Pr i ,left i ,right i ,back i ,r i ,v i}, where Pf i Pb i Pl i Pr iThese are images captured by four fisheye cameras installed on the hood, rear, and under the left and right rearview mirrors of the car. i right i 、back i These are the distances between the car and the obstacle, measured by ultrasonic radars installed on the left, right, and rear sides of the car, respectively; r i v is the steering wheel angle at the time of sampling. i This represents the vehicle's wheel speed at the time of sampling.

[0007] 2) Construct training samples D' using the initial dataset D;

[0008] 3) Build and optimize the CNN-LSTM neural network;

[0009] 4) Train the neural network to obtain an automatic parking driving model;

[0010] 5) Implement automatic parking by using an automatic parking model for real vehicle control.

[0011] Furthermore, step 2) above includes the following steps:

[0012] 2.1) The four vehicle-mounted fisheye cameras were calibrated using Zhang Zhengyou's calibration method to obtain the calibration parameters of the four vehicle-mounted fisheye cameras, including intrinsic and extrinsic parameters;

[0013] 2.2) The fisheye images Pf, Pb, Pl, and Pr are distorted using intrinsic and extrinsic parameters to obtain the distorted transformation images Pf', Pb', Pl', and Pr'.

[0014] 2.3) Transform the correction transformation diagrams Pf', Pb', Pl', and Pr' into top views Pf”, Pb”, Pl”, and Pr”;

[0015] 2.4) Cut and stitch the top views Pf”, Pb”, Pl”, and Pr” to obtain the panoramic stitched view P. O ;

[0016] 2.5) For the panoramic mosaic P O Perform downsampling processing to output an image P of fixed size. T ;

[0017] 2.6) For image P T Normalization is performed to obtain the training image P;

[0018] 2.7) Construct training samples D', denoted as D'={d1',d2'……d i '……},d i 'Includes surround view image frame sequence data, ultrasonic obstacle distance information data, and current driver driving information data, denoted as d'i '={P i ,left i ,right i ,back i ,r i ,v i The training label is denoted as Label = {left} i ,right i ,back i ,r i ,v i}

[0019] Furthermore, step 3) above includes the following steps:

[0020] 3.1) Construct a CNN-LSTM neural network, which includes a CNN, an LSTM, and a feature fusion layer. The CNN part consists of 5 convolutional layers, 5 pooling layers, and 1 fully connected layer; the LSTM part consists of 2 fully connected layers, 1 pooling layer, and 20 LSTM units; the feature fusion layer consists of 1 fusion layer and 2 fully connected layers.

[0021] 3.2) Optimize the neural network using the Adam optimizer.

[0022] Furthermore, step 4) above includes the following steps:

[0023] 4.1) Input training sample D';

[0024] 4.2) Calculate the mean square error (MSE). The formula is as follows:

[0025]

[0026] Pred is the prediction result during training, which is a 2D tensor of the same size as the training label, where i and j are the row and column coordinates, and n is the batch size.

[0027] 4.3) If MSE > mean squared error threshold a, proceed to step 4.1) to continue training; otherwise, proceed to step 5).

[0028] The beneficial effects of this invention are: it solves the problems of existing automatic parking systems where path planning and tracking are cumbersome and inefficient in actual parking calibration work, because there is no skilled driver to operate the parking system and parking control cannot be directly executed according to the parking environment. Attached Figure Description

[0029] Figure 1 This is a flowchart of the automatic parking method based on multi-source information perception and end-to-end deep learning of the present invention. Detailed Implementation

[0030] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. In some instances, means, elements and circuits well known to those skilled in the art are not described in detail in order to highlight the main points of the present invention.

[0031] The embodiments described above are preferred embodiments of the present invention, but the present invention is not limited to the above embodiments. Any obvious improvements, substitutions or modifications that can be made by those skilled in the art without departing from the essence of the present invention shall fall within the protection scope of the present invention.

[0032] like Figure 1 As shown, this invention provides an automatic parking method based on multi-source information perception and end-to-end deep learning, comprising the following steps:

[0033] 1) Sample the parking process at a sampling frequency f to construct an initial dataset D; the initial dataset is denoted as D = {d1, d2, ..., dn}. i ……},d i Let d be the sampled data of the i-th time. i ={Pf i Pb i ,Pl i ,Pr i ,left i ,right i ,back i ,r i ,v i}, where Pf i Pb i Pl i Pr i These are images captured by four fisheye cameras installed on the hood, rear, and under the left and right rearview mirrors of the car. i right i 、back i These are the distances between the car and the obstacle, measured by ultrasonic radars installed on the left, right, and rear sides of the car, respectively; r i v is the steering wheel angle at the time of sampling. i This represents the vehicle's wheel speed at the time of sampling.

[0034] 2) Construct training samples D' using the initial dataset D;

[0035] As a preferred embodiment of the present invention, the following steps are included:

[0036] 2.1) The four vehicle-mounted fisheye cameras were calibrated using Zhang Zhengyou's calibration method to obtain the calibration parameters of the four vehicle-mounted fisheye cameras, including intrinsic and extrinsic parameters;

[0037] 2.2) The fisheye images Pf, Pb, Pl, and Pr are distorted using intrinsic and extrinsic parameters to obtain the distorted transformation images Pf', Pb', Pl', and Pr'.

[0038] 2.3) Transform the correction transformation diagrams Pf', Pb', Pl', and Pr' into top views. First, construct a physical coordinate system with any corner point of the giant chessboard as the origin. Select at least four control points in the chessboard that are not on a straight line and record their actual physical coordinates. Then, find the positions of these control points in the correction transformation diagram of the fisheye image and record their image coordinates. Finally, correlate the physical coordinates and image coordinates of these control points to obtain the homography matrix. Use the homography matrix to transform the correction transformation diagram of the fisheye image into top views Pf”, Pb”, Pl”, and Pr”.

[0039] 2.4) Cut and stitch the top-view transformation images Pf”, Pb”, Pl”, and Pr”. Select areas on the top-view transformation images in the front, back, left, and right directions for cutting to obtain the cut image P. O1 P O2 P O3 P O4 These cutouts are then stitched together to obtain the panoramic stitched image P. O ;

[0040] 2.5) For the panoramic mosaic P O Perform downsampling processing and output an image P with a size of 100*100*3. T ;

[0041] 2.6) For image P T Perform normalization processing and adjust image P. T The saturation, contrast, and brightness of the image are adjusted, and Gaussian noise is used to enhance the image noise, resulting in the training image P.

[0042] 2.7) Construct training samples D', denoted as D'={d1',d2'……d i '……},d i 'Includes surround view image frame sequence data, ultrasonic obstacle distance information data, and current driver driving information data, denoted as d' i '={P i ,left i ,right i ,back i ,r i ,v i The training label is denoted as Label = {left} i ,right i ,back i ,ri ,v i}

[0043] 3) Build and optimize the CNN-LSTM neural network;

[0044] As a preferred embodiment of the present invention, the following steps are included:

[0045] 3.1) Construct a CNN-LSTM neural network, including CNN, LSTM, and feature fusion layer. The CNN part consists of 5 convolutional layers, 5 pooling layers, and 1 fully connected layer; the LSTM part consists of 2 fully connected layers, 1 pooling layer, and 20 LSTM units; the feature fusion layer consists of 1 fusion layer and 2 fully connected layers.

[0046] 3.2) The Adam (adaptive moment estimation) algorithm optimizer is used to accelerate the convergence of the model.

[0047] 4) Train the neural network by inputting the training sample data D' into the constructed deep neural network for training, and obtain the trained end-to-end automatic parking driving model.

[0048] As a preferred embodiment of the present invention, the following steps are included:

[0049] 4.1) Input training sample D';

[0050] Convolutional layer 1: 3*3*3 kernel, stride 1, same padding, ReLU activation;

[0051] Pooling layer 1: 2*2*1, step size 2;

[0052] Convolutional layer 2: 3*3*3 kernel, stride 1, same padding, ReLU activation;

[0053] Pooling layer 2: 2*2*2, step size 2;

[0054] Convolutional layer 3: 3*3*3 kernels, stride 1, same padding, ReLU activation;

[0055] Pooling layer 3: 2*2*2, step size 2;

[0056] Convolutional layer 4: 3*3*3 kernel, stride 1, same padding, ReLU activation;

[0057] Pooling layer 4: 2*2*2, step size 2;

[0058] Convolutional layer 5: 3*3*3 kernel, stride 1, same padding, ReLU activation;

[0059] Pooling layer 5: 2*2*2, step size 2;

[0060] Fully connected layer: 5120 neurons, tanh activation;

[0061] To prevent overfitting, dropout is set with a forgetting rate of 0.1.

[0062] LSTM:

[0063] Fully connected layer 1: 20 neurons, ReLU activated;

[0064] Fully connected layer 2: 50 neurons, ReLU activated;

[0065] LSTM layer: 20 LSTM units, stride 5;

[0066] 4.2) Calculate the mean squared error (MSE), where the formula for calculating MSE is:

[0067]

[0068] Pred is the prediction result during training, which is a 2D tensor of the same size as the training label, where i and j are the row and column coordinates, and n is the batch size.

[0069] 4.3) If MSE > mean square error threshold a, proceed to step 4.1) to continue training; otherwise, if the automatic parking model is obtained, proceed to step 5) to proceed to the next step. In a specific embodiment of the present invention, the mean square error threshold a = 0.005.

[0070] 5) Implement automatic parking by using an automatic parking model for real-world vehicle control:

[0071] Real-time acquisition and input of surround view images and ultrasonic radar distance data through sensors;

[0072] Output the predicted steering wheel angle data r and the vehicle speed data v.

[0073] The end-to-end automated parking model outputs driving commands and completes control based on real-time image information collected by four fisheye cameras and real-time distance information collected by ultrasonic radar. Real-world control results show that the vehicle collision rate is only 2.8% and the parking space deviation rate is only 1.2%.

Claims

1. An automatic parking method based on multi-source information perception and end-to-end deep learning, characterized in that, Includes the following steps: 1) The parking process is sampled at a sampling frequency f to construct an initial dataset D; the initial dataset is denoted as D={d1,d2……d i ...}, d i Let d be the sampled data of the i-th time. i ={Pf i Pb i ,Pl i ,Pr i ,left i ,right i ,back i ,r i ,v i }, where Pf i Pb i Pl i Pr i These are images captured by four fisheye cameras installed on the hood, rear, and under the left and right rearview mirrors of the car. i right i 、back i These are the distances between the car and the obstacle, measured by ultrasonic radars installed on the left, right, and rear sides of the car, respectively; r i v is the steering wheel angle at the time of sampling. i This represents the vehicle's wheel speed at the time of sampling. 2) Construct training samples D' using the initial dataset D; Step 2) includes the following steps: 2.1) The four vehicle-mounted fisheye cameras were calibrated using Zhang Zhengyou's calibration method to obtain the calibration parameters of the four vehicle-mounted fisheye cameras, including intrinsic and extrinsic parameters; 2.2) The fisheye images Pf, Pb, Pl, and Pr are distorted using intrinsic and extrinsic parameters to obtain the distorted transformation images Pf', Pb', Pl', and Pr'. 2.3) Transform the correction transformation diagrams Pf', Pb', Pl', and Pr' into top views Pf'', Pb'', Pl'', and Pr''; 2.4) Cut and stitch the top views Pf'', Pb'', Pl'', and Pr'' to obtain the panoramic stitched view P. O ; 2.5) For the panoramic mosaic P O Perform downsampling processing to output an image P of fixed size. T ; 2.6) For image P T Normalization is performed to obtain the training image P; 2.7) Construct training samples D', denoted as D'={d1',d2'……d i '……},d i 'Includes surround view image frame sequence data, ultrasonic obstacle distance information data, and current driver driving information data, denoted as d' i '={P i ,left i ,right i ,back i ,r i ,v i The training label is denoted as Label={left}. i ,right i ,back i ,r i ,v i }; 3) Build and optimize the CNN-LSTM neural network; Step 3) includes the following steps: 3.1) Construct a CNN-LSTM neural network, which includes a CNN, an LSTM, and a feature fusion layer. The CNN part consists of 5 convolutional layers, 5 pooling layers, and 1 fully connected layer; the LSTM part consists of 2 fully connected layers, 1 pooling layer, and 20 LSTM units; the feature fusion layer consists of 1 fusion layer and 2 fully connected layers. 3.2) Optimize the neural network using the Adam optimizer; 4) Train the neural network to obtain an automatic parking driving model; 5) Implement automatic parking by using an automatic parking model for real vehicle control.

2. The automatic parking method based on multi-source information perception and end-to-end deep learning as described in claim 1, characterized in that, Step 4) includes the following steps: 4.1) Input training sample D'; 4.2) Calculate the mean square error (MSE). The formula is as follows: Pred is the prediction result during training, which is a 2D tensor of the same size as the training label, where i and j are the row and column coordinates, and n is the batch size. 4.3) If MSE > mean squared error threshold a, proceed to step 4.1) to continue training; otherwise, proceed to step 5).

Images