An end-to-end automatic driving method and system under complex road conditions
By combining CNN, spiking neural network and neural circuit network in complex road conditions, the problem of information extraction and computational stability of end-to-end autonomous driving algorithms in complex road conditions is solved, and more efficient and interpretable autonomous driving decision-making is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- AUTOMOBILE RES INST OF TSINGHUA UNIV IN SUZHOU XIANGCHENG
- Filing Date
- 2022-12-29
- Publication Date
- 2026-06-02
AI Technical Summary
Existing end-to-end autonomous driving algorithms struggle to effectively extract information in complex road conditions, and CNN-based methods are computationally intensive and lack interpretability, especially when the data is noisy or interfered with, resulting in unstable predictions.
An end-to-end autonomous driving method for complex road conditions is adopted. A dataset is created by acquiring historical vehicle driving data. High-dimensional feature vectors are extracted using CNN. Then, spiking neural networks and neural circuit network models are combined to perform spiking mapping and prediction. The number and sparsity of neurons are dynamically adjusted to improve robustness and interpretability.
It improves the stability and interpretability of autonomous driving predictions under complex road conditions, reduces computational complexity, and enhances the system's generalization ability and the interpretability of decisions.
Smart Images
Figure CN116331251B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of autonomous driving technology, and in particular to an end-to-end autonomous driving method and system for complex road conditions. Background Technology
[0002] In recent years, Convolutional Neural Networks (CNNs) have achieved tremendous success in the field of computer vision, such as in image classification, object detection, and semantic segmentation. Driven by massive amounts of data, CNNs have achieved surprising results. End-to-end autonomous driving primarily involves the following steps: First, using onboard cameras or radar and data sensors, a large amount of human driving data is collected, including foreground images from the driver's perspective and corresponding control measures taken by the driver, such as steering wheel angle, accelerator, and brake. The foreground images and control information are used as input images and their corresponding labels to create a dataset. Then, algorithms are designed to learn and summarize from this dataset. Finally, on new roads or with new foreground images as input, predictions of appropriate control measures are provided.
[0003] Currently, most end-to-end autonomous driving algorithms are based on CNNs. Their massive computational demands and lack of interpretability are precisely what autonomous driving requires. Furthermore, while CNNs can achieve good predictive control results under ideal input data conditions, they become highly unstable when the data is noisy or interfered with, or when the input is damaged, such as by sudden direct sunlight.
[0004] However, end-to-end autonomous driving in complex road conditions has a more complex task, and simple CNNs cannot extract more useful information. 19 LTC control neurons cannot cope with more complex high-dimensional information. Summary of the Invention
[0005] In view of this, the present invention proposes an end-to-end autonomous driving method and system for complex road conditions, in order to solve the problem of autonomous driving under complex road conditions.
[0006] The first aspect of this invention provides an end-to-end autonomous driving method for complex road conditions. The method includes: acquiring historical vehicle driving data under complex road conditions and creating a dataset; dividing the dataset into a training set and a test set; inputting the training set into a feature extraction network to obtain high-dimensional feature vectors; establishing a spiking neural network and performing spiking mapping on the extracted high-dimensional feature vectors; inputting the spiking-mapped high-dimensional feature vectors into a neural circuit network model to obtain predicted values; calculating the loss between the predicted values and the test set; adjusting the training parameters in the neural circuit network model based on the loss and training the neural circuit network model to obtain a prediction model; acquiring real-time images of the vehicle ahead under complex road conditions, inputting the real-time images into the prediction model to obtain steering wheel angle, vehicle speed, accelerator pedal depth, and brake pedal depth information, and using the steering wheel angle, vehicle speed, accelerator pedal depth, and brake pedal depth information as the decision result.
[0007] Furthermore, the historical vehicle driving data includes historical images of the vehicle ahead and corresponding control information; the control information includes at least data on steering wheel angle, vehicle speed, accelerator pedal depth, and brake pedal depth under complex road conditions.
[0008] Furthermore, a dataset is created using historical vehicle front images and corresponding control information; the historical vehicle front images in the dataset are divided into a training set, and the control information corresponding to the historical vehicle front images in the dataset is divided into a test set.
[0009] Furthermore, the neural circuit network model includes a perception layer, an intermediate layer, a command layer, and an action layer. The perception layer is used to receive high-dimensional feature vectors after pulse mapping. The intermediate layer and the command layer are used to process the high-dimensional feature vectors. The action layer is used to output predicted values.
[0010] Furthermore, the method for calculating the loss between the predicted value and the test set is as follows:
[0011]
[0012] Where n refers to the number of datasets, y pre This refers to the predicted value of the control information, y real This refers to the control information in the test set.
[0013] Furthermore, the step of adjusting the training parameters in the neural circuit network model based on the loss and training the neural circuit network model to obtain a prediction model includes: feeding the loss back into the neural circuit network model and adjusting the training parameters in the neural circuit network model; training the neural circuit network model based on the loss until the loss is lower than a preset threshold, then stopping the training; and determining the neural circuit network model containing the training parameters as the prediction model.
[0014] A second aspect of the present invention provides an end-to-end autonomous driving system for complex road conditions, the system comprising: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of the end-to-end autonomous driving method for complex road conditions as described above.
[0015] The aforementioned end-to-end autonomous driving method and system for complex road conditions solves the biological uninterpretability of CNN-based algorithms in complex road conditions. Furthermore, it allows for the selection of different feature extraction networks and neural circuit network models with different numbers of neurons and different sparsity based on the complexity of the road conditions. Attached Figure Description
[0016] For illustrative and not limiting purposes, the invention will now be described with reference to preferred embodiments thereof, particularly the accompanying drawings, in which:
[0017] Figure 1 This is a flowchart of an end-to-end autonomous driving method under complex road conditions provided in an embodiment of this application;
[0018] Figure 2 This is a schematic diagram of the PINet network selected in this application embodiment performing feature extraction on the input image;
[0019] Figure 3 This is a schematic diagram showing the feature map results of the features extracted by the PINet network selected in the embodiments of this application;
[0020] Figure 4 This is a network architecture diagram provided in an embodiment of this application;
[0021] Figure 5 This is a schematic diagram of an end-to-end autonomous driving system under complex road conditions provided in an embodiment of this application. Detailed Implementation
[0022] To better understand the above-mentioned objects, features, and advantages of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments of the present invention and the features thereof can be combined with each other.
[0023] Numerous specific details are set forth in the following description to provide a thorough understanding of the invention. The described embodiments are merely some, not all, of the embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.
[0024] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.
[0025] The following describes in detail, with reference to the accompanying drawings, an end-to-end autonomous driving method for complex road conditions provided by an embodiment of this application.
[0026] Please see Figure 1 The end-to-end autonomous driving methods for this complex road condition include:
[0027] Step 101: Obtain historical vehicle driving data under complex road conditions and create a dataset.
[0028] In this embodiment of the application, the historical vehicle driving data includes a historical image of the vehicle in front and corresponding control information. The control information includes at least steering wheel angle, vehicle speed, accelerator pedal depth, and brake pedal depth data.
[0029] In this embodiment, an in-vehicle camera captures images of the vehicle's front under complex road conditions. LiDAR and sensors are used to collect data such as steering wheel angle, vehicle speed, accelerator pedal depth, and brake pedal depth under these conditions.
[0030] In some embodiments, historical images of the vehicle's front are used as input images, and data such as steering wheel angle, vehicle speed, accelerator pedal depth, and brake pedal depth are used as labels corresponding to the input images. A dataset is created based on the input images and their corresponding labels. The historical images of the vehicle's front in the dataset are divided into a training set, and the control information corresponding to the historical images of the vehicle's front in the dataset is divided into a test set.
[0031] Step 102: Input the historical vehicle front images in the dataset into the feature extraction network to obtain the corresponding high-dimensional feature vectors.
[0032] In this embodiment, a CNN network is used as the feature extraction network. The images of the vehicles in front of the dataset are input into the feature extraction network to extract the corresponding high-dimensional feature vectors.
[0033] In this embodiment, the CNN network used for feature extraction can be any network that performs best in computer vision tasks, such as ResNet, RetinaNet, PINet, and HRNet. Only the feature extraction part is retained, and the final decision part is discarded.
[0034] Figure 2 This is a schematic diagram illustrating the PINet network used in this embodiment of the application to extract features from an input image. Please refer to [link / reference]. Figure 2The PINet network mainly consists of five convolutional layers. The Resizing Layer comprises three convolutional blocks with a stride of 2, batch normalization, and activation operations, achieving the goal of reducing the size of the original large image to extract features. The output image size is 80*64 pixels with 128 channels. Layer 1 is a funnel-shaped structure in the PINet network, mainly composed of four downsampling convolutional layers and four upsampling convolutional layers. After passing through this layer, the input size and number of channels do not change. In this embodiment, the smallest feature map of the fourth downsampling is retained, with a size of 5*4 pixels and 128 channels. Layers 2, 3, and 4, like Layer 1, are funnel-shaped convolutional modules and all retain the smallest feature map. The four retained feature maps are concatenated according to the number of channels to form a 5*4 pixel feature map with 512 channels. Then, a convolutional layer is passed to form a 5*4 pixel feature map with 128 channels. Finally, a straightening layer is passed to form a 2560 vector. After passing through a spiking neural network for spiking mapping to reduce the amount of computation, the vector is then input into a neural circuit network for prediction.
[0035] Figure 3 This is a schematic diagram showing the feature map display results of the features extracted by the PINet network selected in this embodiment of the application. In this embodiment, a random input image was selected, and 20 feature maps were chosen and enlarged to 640*412 pixels to form the feature map. It can be seen that as the number of network layers increases, the feature map encompasses more and more information, and the feature map becomes more abstract.
[0036] Step 103: Establish a spiking neural network and use the spiking neural network to perform pulse mapping on the high-dimensional feature vector extracted in step 102.
[0037] In this embodiment, a spiking neural network (SNN) is established using the high-dimensional feature vectors obtained from the feature extraction network.
[0038] In this embodiment, a spiking neural network (SNN) is used to perform spiking mapping on the high-dimensional feature vector extracted in step 102, reducing the computational load. As a third-generation neural network, the SNN uses a spiking neuron model to simulate and explain the information processing process of biological neurons.
[0039] The high-dimensional feature vector extracted in step 102 is input into the spiking neural network (SNN). After processing by the spiking neuron model in the SNN, the feature vector after pulse mapping is obtained.
[0040] Step 104: Input the high-dimensional feature vector after pulse mapping into the neural circuit network model to obtain the predicted value of the control information.
[0041] In this embodiment, a Neural Circuit Policies (NCP) model is used to predict the control information of the input image. The main function of the Neural Circuit Policies (NCP) is to learn to map high-dimensional inputs into steering commands. It has 19 control neurons and 253 synaptic neurons, and realizes the determination of steering commands from 32 encapsulated high-dimensional input features.
[0042] Figure 4 This is a network architecture diagram provided in an embodiment of this application. Please refer to... Figure 4 The Neural Circuit Network (NCP) model mainly comprises four hierarchical layers: a Sensory Neuron, an InterNeuron, a Command Neuron, and a Montor Neuron. The Sensory Neuron receives high-dimensional feature vectors after impulse mapping; the InterNeuron and Command Neurons process these high-dimensional feature vectors; and the Montor Neuron outputs predicted values for control information. In this embodiment, the signal transmission mechanism of the Sensory, InterNeuron, Command, and Montor Neurons in the NCP model is based on LTC (liquid time constant) neurons. The neural dynamics are given by continuous ordinary differential equations, exhibiting a nonlinear time-varying synaptic transmission mechanism, which greatly enriches the expressive power of time series. Compared to black-box learning CNNs, which are orders of magnitude larger, this network demonstrates superior generalization, interpretability, and robustness. The obtained neural agent enables task-specific parts of complex autonomous systems to possess high-fidelity autonomy.
[0043] The number of neurons in the NCP network is dynamically adjusted based on the complexity of the road condition data, and its sparsity can also be dynamically adjusted until the ideal decision result is achieved. The neuron transmission method uses LTC neurons, with each neuron having x... i The state dynamics of (t) when linked to neuron j via a neural synapse are as follows:
[0044]
[0045] in, It has leakage conductivity The time constant of neuron i, w ij These are the prominent weights of neurons i to j. It is a modular capacitor. It is the resting potential, E ij It is a reverse synaptic potential that defines the salient polarity.
[0046] Step 105: Calculate the loss between the predicted value of the control information in step 104 and the control information in the dataset in step 101.
[0047] In this embodiment, the control information in the dataset includes steering wheel angle, vehicle speed, accelerator pedal depth, and brake pedal depth, which are presented numerically. Therefore, this embodiment uses mean squared error to calculate the loss between the predicted value of the control information and the control information in the dataset. The loss is calculated as follows:
[0048]
[0049] Where n refers to the number of datasets, y pre This refers to the predicted value of the control information, y real It is the control information in the dataset.
[0050] Step 106: Adjust the training parameters in the neural circuit network model according to the loss and train the neural circuit network model to obtain the prediction model.
[0051] In some embodiments, adjusting the training parameters in the neural circuit network model based on the loss and training the neural circuit network model to obtain the prediction model includes:
[0052] The loss is fed back into the neural circuit network model, and the training parameters in the neural circuit network model are adjusted.
[0053] The neural circuit network model is trained based on the loss until the loss is lower than a preset threshold, at which point training stops.
[0054] The neural circuit network model containing training parameters is identified as the prediction model.
[0055] In this embodiment of the application, during the training process, when the loss meets a preset requirement, the neural circuit network model can be considered to have high performance and meet the usage requirements. For example, a preset threshold can be set. When the loss value is less than the preset threshold, the requirement is met, training ends, and the neural circuit network model containing the training parameters is determined as the prediction model.
[0056] In this embodiment, after calculating the loss between the predicted value of the control information and the control information in the dataset, backpropagation is used to update the parameters of the neural circuit network model to obtain the optimal prediction model.
[0057] Step 107: Obtain real-time images of the vehicle ahead under complex road conditions, input the real-time images of the vehicle ahead into the prediction model to obtain information on steering wheel angle, vehicle speed, accelerator pedal depth and brake pedal depth, and use the information on steering wheel angle, vehicle speed, accelerator pedal depth and brake pedal depth as the decision result.
[0058] The aforementioned end-to-end autonomous driving method for complex road conditions solves the biological uninterpretability of CNN-based algorithms in complex road conditions. Furthermore, it allows for the selection of different feature extraction networks and neural circuit network models with different numbers of neurons and different sparsity based on the complexity of the road conditions.
[0059] The aforementioned end-to-end autonomous driving method for complex road conditions combines the powerful feature extraction performance of CNNs with the temporal characteristics of RNNs. Furthermore, the RNN neurons are based on the LTC neuron model, mimicking the biomimetic network wiring diagram to build a network with better biological interpretability and sparsity, eliminating the adverse effects of the temporal characteristics of RNNs. Additionally, the use of spiking neural networks (SNNs) reduces computational complexity and improves computational speed.
[0060] For the method embodiments described above, please refer to [link / reference]. Figure 5 This application also provides an end-to-end autonomous driving system for complex road conditions. The system 200 may include:
[0061] Memory 201 is used to store computer programs;
[0062] When processor 202 executes the computer program stored in memory 201, it can perform the following steps:
[0063] Historical vehicle driving data under complex road conditions is acquired and a dataset is created. The dataset is divided into a training set and a test set. The training set is input into a feature extraction network to obtain high-dimensional feature vectors. A spiking neural network is established, and the extracted high-dimensional feature vectors are pulse-mapped using the spiking neural network. The pulse-mapped high-dimensional feature vectors are input into a neural circuit network model to obtain predicted values. The loss between the predicted values and the test set is calculated. The training parameters in the neural circuit network model are adjusted according to the loss, and the neural circuit network model is trained to obtain a prediction model. Real-time images of the vehicle ahead under complex road conditions are acquired and input into the prediction model to obtain information on steering wheel angle, vehicle speed, accelerator pedal depth, and brake pedal depth. The steering wheel angle, vehicle speed, accelerator pedal depth, and brake pedal depth information are used as the decision results.
[0064] For a description of the apparatus provided in the embodiments of this application, please refer to the above method embodiments; further details will not be repeated here.
[0065] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.
Claims
1. An end-to-end autonomous driving method for complex road conditions, characterized in that, include: Acquire historical vehicle driving data under complex road conditions and create a dataset; The dataset is divided into a training set and a test set; The training set is input into the feature extraction network to obtain high-dimensional feature vectors; A spiking neural network is established, and the extracted high-dimensional feature vectors are pulse-mapped using the spiking neural network. The high-dimensional feature vector after pulse mapping is input into the neural circuit network model to obtain the predicted value; Calculate the loss between the predicted values and the test set; Adjust the training parameters in the neural circuit network model according to the loss and train the neural circuit network model to obtain the prediction model; The system acquires real-time images of the vehicle ahead under complex road conditions, inputs these images into a prediction model to obtain information on steering wheel angle, vehicle speed, accelerator pedal depth, and brake pedal depth, and uses this information as the decision result. The neural circuit network model includes a perception layer, an intermediate layer, a command layer, and an action layer. The perception layer is used to receive high-dimensional feature vectors after pulse mapping. The intermediate layer and the command layer are used to process the high-dimensional feature vectors. The action layer is used to output predicted values. The perception layer, intermediate layer, command layer, and action layer in the neural circuit network model transmit signals through LTC neurons.
2. The end-to-end autonomous driving method under complex road conditions according to claim 1, characterized in that, The historical vehicle driving data includes historical images of the vehicle ahead and corresponding control information; the control information includes at least data on steering wheel angle, vehicle speed, accelerator pedal depth, and brake pedal depth under complex road conditions.
3. The end-to-end autonomous driving method under complex road conditions according to claim 2, characterized in that, A dataset was created using historical vehicle front images and corresponding control information. The historical vehicle front images in the dataset were divided into a training set, and the corresponding control information in the dataset was divided into a test set.
4. The end-to-end autonomous driving method under complex road conditions according to claim 3, characterized in that, The method for calculating the loss between the predicted value and the test set is as follows: Where n refers to the number of datasets, This refers to the predicted value of control information. This refers to the control information in the test set.
5. The end-to-end autonomous driving method under complex road conditions according to claim 1, characterized in that, The step of adjusting the training parameters in the neural circuit network model based on the loss and training the neural circuit network model to obtain the prediction model includes: The loss is fed back into the neural circuit network model, and the training parameters in the neural circuit network model are adjusted. The neural circuit network model is trained based on the loss until the loss is lower than a preset threshold, at which point training stops. The neural circuit network model containing the training parameters is identified as the prediction model.
6. An end-to-end autonomous driving system for complex road conditions, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the end-to-end autonomous driving method for complex road conditions as described in any one of claims 1 to 5 when executing the computer program.