Lane line unsupervised keeping method based on brain-like intelligence

Abstract

The invention provides a lane line unsupervised keeping method based on brain-like intelligence, and the method comprises the steps: obtaining RGBD four-channel data through a monocular camera and a depth camera, carrying out the standardization processing, and inputting a model composed of a convolution layer and a brain-like layer. The convolutional layer extracts data features, and the brain-like layer simulates a biological neuron working mechanism to output a vehicle steering angle. The model is pre-trained by using a Comma. Ai Driving Dataset data set, then is subjected to unsupervised training through a self-collected RGBD data set, and is optimized by using MSE Loss as a loss function and MAE as an evaluation index. The network constructed by the method is small in scale, few in parameters, high in interpretability, high in robustness in different noise environments, and remarkable in calculation efficiency and resource consumption. The method has been successfully applied to park inspection robots to realize autonomous navigation obstacle avoidance, can also be transplanted to a vehicle-mounted chip platform, and has a wide application prospect in the field of automatic driving.

Description

Technical Field

[0001] The present invention relates to the field of brain-inspired intelligence and artificial intelligence technology, and in particular to an unsupervised lane keeping method based on brain-inspired intelligence. Background Art

[0002] Currently, lane keeping technology, a core function of autonomous driving, relies primarily on multi-sensor fusion and supervised learning frameworks. Its technical approach can be broadly divided into three areas: vision-based detection and tracking, multimodal sensor fusion, and path planning and control strategies.

[0003] In the field of visual inspection, traditional methods rely on computer vision algorithms to process road images captured by on-board cameras. For example, they use edge detection algorithms like the Canny operator combined with the Hough transform to identify lane geometry, or employ sliding window search to fit lane curvature. With the rise of deep learning, convolutional neural networks (CNNs) and Transformer architectures have become increasingly mainstream. Mobileye's EyeQ series chips, for example, utilize pre-trained deep neural networks to output lane position and curvature parameters in real time. However, these models rely heavily on supervised training using large-scale annotated datasets (such as BDD100K and ApolloScape). This annotation process is not only costly but also faces challenges in consistency and coverage. Different annotators may have different criteria for determining blurred lane lines or temporary markings, resulting in noisy training data. More critically, performance degrades dramatically when the model encounters scenarios not covered by the training data, such as extreme weather conditions, complex lighting conditions, or unusual road markings. Re-collecting and annotating data requires significant resources, creating a vicious cycle. In a global market, lane design, road materials, and traffic regulations vary significantly across regions, making it difficult for a single model to achieve good generalization.

[0004] Multimodal sensor fusion solutions are designed to improve robustness in complex scenarios. Taking Tesla's Autopilot system as an example, it uses cameras as the main sensor, supplemented by millimeter-wave radar data. Through spatiotemporal alignment and probabilistic model fusion, it reduces the false detection rate of a single sensor in low-light, rainy and snowy weather, or lane line wear scenarios. Some high-end models further introduce laser radar (LiDAR) point cloud data, using its high-precision three-dimensional perception capabilities to assist in lane line positioning. However, such systems need to solve the technical challenges brought about by multi-sensor calibration (such as time synchronization and coordinate alignment) and data heterogeneity (such as modal differences between images and point clouds), and the hardware cost is relatively high, which limits its widespread application in mass-produced models.

[0005] Path planning and control strategies are the final execution links of lane keeping. Based on the detected lane information, the vehicle needs to generate lateral control instructions (such as steering angle or torque) to ensure that it drives along the center of the lane. Traditional methods use PID controllers or model predictive control (MPC). The former relies on experience-based parameter adjustment, while the latter optimizes future multi-step control sequences by building a vehicle dynamics model. In recent years, reinforcement learning (RL) has been introduced into control strategy design, which learns adaptive strategies through interaction with the simulation environment. For example, Bosch's lane keeping assist system (LKAS) combines MPC with vehicle state feedback to dynamically adjust the steering torque. However, such methods are highly dependent on accurate vehicle dynamics model parameters (such as center of mass position and tire friction coefficient), and are prone to control deviations on complex roads or when vehicle load changes.

[0006] Overall, while existing technologies have achieved high-precision lane detection and tracking, their core reliance on manually annotated data, costly sensors, and static control models severely restricts large-scale applications and the ability to generalize across scenarios. Therefore, an innovative approach is urgently needed to overcome these bottlenecks and achieve more efficient, reliable, and cost-effective autonomous driving solutions. Summary of the Invention

[0007] The present invention provides an unsupervised lane keeping method based on brain-like intelligence to address the many defects of existing lane keeping technology in terms of technical architecture and implementation methods, and achieve more efficient, reliable and low-cost autonomous driving.

[0008] The present invention provides an unsupervised lane keeping method based on brain-like intelligence, comprising the following steps:

[0009] A. Data acquisition: Use a monocular camera to capture the color image of the scene, and a depth camera to obtain the image depth information to obtain RGBD four-channel data;

[0010] B. Data preprocessing: For the RGBD four-channel data, different means and standard deviations are used for standardization. The RGB image is processed according to a specific formula, and the D image is also processed according to the corresponding formula after being mapped to the three RGB channels;

[0011] C. Feature Extraction and Decision-Making: The preprocessed data is input into a model consisting of convolutional layers and brain-inspired layers. The convolutional layer consists of four blocks. The first three blocks extract features through different convolutional layers and transmit them to the fully connected layer. The brain-inspired layer consists of 19 internal neurons and 64 sensory neurons. The neurons follow a specific state update formula and learn mapping relationships by adjusting synaptic weights, ultimately outputting the vehicle's steering angle.

[0012] D. Model training: We first pre-trained the model using the Comma.ai Driving Dataset, then performed unsupervised training on a self-collected RGBD dataset. We used the Adam algorithm with a learning rate of 1e-5 to optimize model parameters, set the batch size to 32, the sequence to 16, and trained for 3500 epochs.

[0013] E. Model evaluation: Using MSE Loss as the loss function and MAE as the evaluation indicator, the final output of the model is [batch-size, sequence, 1] to predict the steering wheel angle of each frame image.

[0014] Preferably, when the RGB image is normalized, for each pixel value (R, G, B) in the RGB image, the normalized pixel value is calculated as follows:

[0015]

[0016] Preferably, when the D image is normalized, for each pixel value (R, G, B) mapped to the three RGB channels, the normalized pixel value is calculated as follows:

[0017]

[0018] Preferably, the state update formula of neurons in the brain-like layer is:

[0019] v i (t+1)=f(v i (t),I i (t))

[0020] Among them, v i (t) is the state of neuron i at time t, I i (t) is the input of neuron i at time t, and f is the state update function, which is usually a nonlinear function.

[0021] Preferably, the brain-like layer outputs a steering command, which indicates the steering angle of the vehicle. The steering command is calculated through the discharge activity of neurons and can be specifically expressed as:

[0022]

[0023] Here, g is an output function, usually a linear or nonlinear function, used to convert the firing activity of neurons into actual steering commands.

[0024] Preferably, the output of the final model is [batch-size, sequence, 1], and the steering wheel angle of each frame is predicted. The loss adopts MSE Loss, and the calculation formula is as follows:

[0025]

[0026] The calculation formula of the evaluation index MAE is as follows:

[0027] Preferably, an unsupervised lane keeping system based on brain-like intelligence includes:

[0028] Data acquisition unit: consists of a monocular camera and a depth camera, used to obtain RGBD four-channel data;

[0029] Data processing unit: standardize and preprocess the collected RGBD data;

[0030] Model operation unit: Contains convolutional layers and brain-like layers, performs feature extraction, learning and decision-making, and outputs steering angles;

[0031] Training unit: Use specified data sets and algorithms to perform pre-training and unsupervised training on the model;

[0032] Evaluation unit: Evaluate the model based on MSE Loss and MAE.

[0033] Preferably, an electronic device is characterized in that it includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of an unsupervised lane keeping method based on brain-like intelligence are implemented.

[0034] Beneficial effects:

[0035] (1) Compared with traditional CNN and LSTM-based neural networks, the scale of the autonomous driving network constructed by the method of the present invention is significantly reduced. The minimalist autonomous driving system consisting of 19 internal neurons uses only tens of thousands of training parameters, which is 2 orders of magnitude lower than the number of parameters, effectively reducing the amount of computation and storage requirements.

[0036] (2) In the present invention, the interpretability of the brain-like layer is significantly enhanced due to the small number of neurons in the layer. It is possible to clearly observe which parts of the image the network focuses on, such as the roadside or the horizon, and to understand the role of each neuron in driving decisions, which facilitates analysis and optimization of the model.

[0037] (3) Under input noise of different variances, the model of the present invention shows strong resistance to input artifacts, which indicates that the model can focus on key information on the road more stably when facing noise interference.

[0038] (4) The proposed model is comparable to previous state-of-the-art models (CNN, CT-RNN, LSTM) in terms of training squared error and test squared error. However, the control network is 970 times sparser than the LSTM and 241 times sparser than the CT-RNN. This gives the proposed model a significant advantage in computational efficiency and resource consumption, making it more suitable for running on in-vehicle devices.

[0039] The above description is only an overview of the technical solutions of the embodiments of the present invention. In order to more clearly understand the technical means of the embodiments of the present invention, they can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the embodiments of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are specifically listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following is a brief introduction to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0041] Figure 1 It is a flowchart of the present invention;

[0042] Figure 2 This is the network structure diagram of the algorithm of the present invention;

[0043] Figure 3 This is a block diagram of the unsupervised system principle of the present invention;

[0044] Figure 4 This is a straight-line driving effect diagram of the present invention;

[0045] Figure 5 This is a rendering of the present invention's driving effect on a curve;

[0046] Figure 6 This is a rendering of an embodiment of the present invention;

[0047] Explanation of the accompanying symbols: data acquisition unit 1, data processing unit 2, model calculation unit 3, training unit 4, evaluation unit 5. DETAILED DESCRIPTION

[0048] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0049] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0050] See also Figures 1-6 The present invention discloses an unsupervised lane keeping method based on brain-like intelligence, comprising the following steps:

[0051] A. Data Acquisition: A monocular camera captures a color image of the scene, and a depth camera acquires image depth information, resulting in RGBD four-channel data. The monocular camera provides a color image of the scene, suitable for daily visual observation and image processing. The depth camera captures image depth information, helping the model better understand the distance to the target object. The combination of the two provides the model with RGBD four-channel data to achieve lane-keeping autonomous driving.

[0052] B. Data preprocessing: For the RGBD four-channel data, different means and standard deviations are used for standardization. The RGB image is processed according to a specific formula, and the D image is also processed according to the corresponding formula after being mapped to the three RGB channels;

[0053] C. Feature extraction and decision-making: The preprocessed data is input into a model consisting of a convolutional layer and a brain-like layer. The convolutional layer contains 4 blocks. The first three blocks extract features through different convolutional layers and transmit them to the fully connected layer. The RGBD four-channel data is subjected to feature extraction through convolution operations, and the extracted features are sent to the final fully connected layer. These 4 blocks are denoted as C1, C2, C3, and F4 respectively; the convolutional layer can fully integrate multimodal complementarity and provide effective feature information for the subsequent brain-like layer; the brain-like layer consists of 19 internal neurons and 64 sensory neurons. The neurons follow a specific state update formula and learn the mapping relationship by adjusting the synaptic weights, ultimately outputting the vehicle's steering angle;

[0054] D. Model Training: Pre-training was performed using the Comma.ai Driving Dataset, which consists of dashcam video footage of vehicles starting from the same location, driving along different routes, and returning to the starting point at different time periods. Steering angles were also recorded during driving. Stopping and reversing situations were removed, and normal forward driving data was retained for pre-training. Unsupervised training was then performed on a self-collected RGBD dataset. Model parameters were optimized using the Adam algorithm with a learning rate of 1e-5. The batch size was set to 32, the sequence to 16, and training was performed for 3500 epochs.

[0055] E. Model evaluation: Using MSE Loss as the loss function and MAE as the evaluation indicator, the final output of the model is [batch-size, sequence, 1] to predict the steering wheel angle of each frame image.

[0056] In the present invention, when the RGB image is normalized, for each pixel value (R, G, B) in the RGB image, the normalized pixel value is calculated as follows:

[0057]

[0058] In the present invention, when the D image is normalized, for each pixel value (R, G, B) mapped to the three RGB channels, the normalized pixel value is calculated as follows:

[0059]

[0060] In the present invention, the state update formula of neurons in the brain-like layer is:

[0061] v i (t+1)=f(v i (t),I i (t))

[0062] Among them, v i (t) is the state of neuron i at time t, I i (t) is the input of neuron i at time t, and f is the state update function, which is usually a nonlinear function.

[0063] In the present invention, the brain-like layer outputs a steering command, which represents the steering angle of the vehicle. The steering command is calculated through the discharge activity of neurons and can be specifically expressed as:

[0064]

[0065] Here, g is an output function, usually a linear or nonlinear function, used to convert the firing activity of neurons into actual steering commands.

[0066] In this invention, the output of the final model is [batch-size, sequence, 1], and the steering wheel angle of each frame is predicted. The loss adopts MSE Loss, and the calculation formula is as follows:

[0067]

[0068] The calculation formula of the evaluation index MAE is as follows:

[0069] The present invention also discloses an unsupervised lane keeping system based on brain-like intelligence, comprising:

[0070] Data acquisition unit 1: consists of a monocular camera and a depth camera, used to obtain RGBD four-channel data;

[0071] Data processing unit 2: standardize and preprocess the collected RGBD data;

[0072] Model operation unit 3: Contains convolutional layers and brain-like layers, performs feature extraction, learning and decision-making, and outputs the steering angle;

[0073] Training Unit 4: Pre-training and unsupervised training of the model using specified datasets and algorithms;

[0074] Evaluation Unit 5: Evaluate the model based on MSE Loss and MAE.

[0075] In addition, the present invention also discloses an electronic device, characterized in that it includes a memory, a processor, and a computer program stored in the memory and runnable on the processor, and when the processor executes the computer program, it implements the steps of an unsupervised lane keeping method based on brain-like intelligence.

[0076] Example:

[0077] The algorithm of this invention has been successfully transplanted to the park inspection robot, realizing autonomous navigation and obstacle avoidance functions within the park. The specific implementation process is as follows:

[0078] Model deployment: Convert the trained PyTorch model to NCNN format and deploy it on the campus inspection robot. NCNN is a high-performance neural network forward computing framework optimized for mobile devices. It can calculate the robot's steering angle in real time on the robot's CPU, meeting the robot's real-time performance requirements.

[0079] Data Collection and Training: The training data uses RGBD data streams, collected by the robot while driving within the park, and combined with the Comma.ai Driving Dataset for pre-training and unsupervised training. During training, model parameters are continuously optimized to improve the model's adaptability to the park's road scenes.

[0080] Actual Operational Results: After actual operational testing, the campus inspection robot was able to accurately identify lane markings on campus roads under varying lighting and environmental conditions, achieving stable autonomous navigation and obstacle avoidance. During training, the changes in training loss were recorded. These operational results demonstrate the feasibility and effectiveness of the proposed algorithm in practical applications, providing a reference for the application of autonomous driving technology in other fields.

[0081] In addition, the unsupervised autonomous driving algorithm provided by the present invention can be quickly transplanted to various domestic and foreign vehicle-mounted chip platforms, and realize all-weather lane line autonomous driving tasks, especially in low light, rain and fog and other harsh environments.

[0082] Lane keeping technology is a core feature of autonomous driving (Levels 2-4) on highways and urban roads. Through unsupervised learning, vehicles can identify lane markings in real time without relying on high-precision annotated data, adapting to road marking standards (such as dashed, solid, and double yellow lines) in different countries. Traditional vision algorithms are prone to failure at night or in extreme weather conditions (such as heavy rain and snow). Unsupervised models can leverage the vehicle's own motion data (such as steering angle and wheel speed) for continuous optimization and enhanced robustness.

[0083] The advantage of unsupervised technology is that there is no need to repeatedly label data for different vehicle models. The model can be iterated through unlabeled driving data, without the need for complex and arduous data collection and labeling processes. Moreover, since the model itself has very small number of parameters, it can be quickly iterated and deployed.

[0084] In summary, compared with traditional CNN- and LSTM-based neural networks, the scale of the autonomous driving network constructed by the method of the present invention is significantly reduced. The minimalist autonomous driving system consisting of 19 internal neurons uses only tens of thousands of training parameters, reducing the number of parameters by two orders of magnitude, effectively reducing the amount of computation and storage requirements. In the present invention, due to the smaller number of neurons in the brain-like layer, its interpretability is significantly enhanced. It is possible to clearly observe which parts of the image the network focuses on, such as the roadside or the horizon, and to understand the role of each neuron in driving decisions, facilitating model analysis and optimization. Under input noise of different variances, the model of the present invention demonstrates strong resistance to input artifacts, indicating that the model can more stably focus on key information on the road in the face of noise interference. The model of the present invention is comparable to previous SOTA models (CNN, CT-RNN, LSTM) in terms of training squared error and test squared error. However, the control network is 970 times sparser than LSTM and 241 times sparser than CT-RNN, which gives the model of the present invention significant advantages in computational efficiency and resource consumption, making it more suitable for running on vehicle-mounted equipment.

[0085] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions described in the above embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An unsupervised lane keeping method based on brain-inspired intelligence, characterized by: The following steps are involved: A. Data acquisition: Use a monocular camera to capture the color image of the scene, and a depth camera to obtain the image depth information to obtain RGBD four-channel data; B. Data preprocessing: For the RGBD four-channel data, different means and standard deviations are used for standardization. The RGB image is processed according to a specific formula, and the D image is also processed according to the corresponding formula after being mapped to the three RGB channels; C. Feature Extraction and Decision-Making: The preprocessed data is input into a model consisting of convolutional layers and brain-inspired layers. The convolutional layer consists of four blocks. The first three blocks extract features through different convolutional layers and transmit them to the fully connected layer. The brain-inspired layer consists of 19 internal neurons and 64 sensory neurons. The neurons follow a specific state update formula and learn mapping relationships by adjusting synaptic weights, ultimately outputting the vehicle's steering angle. D. Model training: We first pre-trained the model using the Comma.ai Driving Dataset, then performed unsupervised training on a self-collected RGBD dataset. We used the Adam algorithm with a learning rate of 1e-5 to optimize model parameters, set the batch size to 32, the sequence to 16, and trained for 3500 epochs. E. Model evaluation: Using MSE Loss as the loss function and MAE as the evaluation indicator, the final output of the model is [batch-size, sequence, 1] to predict the steering wheel angle of each frame image.

2. The unsupervised lane keeping method based on brain-inspired intelligence according to claim 1, characterized in that: During the RGB image normalization process, for each pixel value (R, G, B) in the RGB image, the normalized pixel value is calculated as follows:

3. The unsupervised lane keeping method based on brain-inspired intelligence according to claim 1, characterized in that: When the D image is normalized, for each pixel value (R, G, B) mapped to the three RGB channels, the normalized pixel value is calculated as follows:

4. The unsupervised lane keeping method based on brain-inspired intelligence according to claim 1, characterized in that: The state update formula of neurons in the brain-like layer is: v i (t+1)=f(v i (t),I i (t)) Among them, v i (t) is the state of neuron i at time t, I i (t) is the input of neuron i at time t, and f is the state update function, which is usually a nonlinear function.

5. The unsupervised lane keeping method based on brain-inspired intelligence according to claim 1, characterized in that: The output of the brain-like layer is a steering command, which represents the steering angle of the vehicle. This steering command is calculated through the discharge activity of neurons and can be expressed as: Here, g is an output function, usually a linear or nonlinear function, used to convert the firing activity of neurons into actual steering commands.

6. The unsupervised lane keeping method based on brain-inspired intelligence according to claim 5, characterized in that: The output of the final model is [batch-size, sequence, 1], and the steering wheel angle of each frame is predicted. The loss uses MSELoss, and the calculation formula is as follows: The calculation formula of the evaluation index MAE is as follows:

7. An unsupervised lane keeping system based on brain-inspired intelligence, used to implement the method according to any one of claims 1 to 6, characterized in that: include: Data acquisition unit (1): consists of a monocular camera and a depth camera, used to obtain RGBD four-channel data; Data processing unit (2): performs standardized preprocessing on the collected RGBD data; Model operation unit (3): Contains convolutional layers and brain-like layers, performs feature extraction, learning and decision-making, and outputs the steering angle; Training unit (4): pre-training and unsupervised training of the model using the specified dataset and algorithm; Evaluation unit (5): Evaluate the model based on MSE Loss and MAE.

8. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method implements the steps of the unsupervised lane keeping method based on brain-like intelligence as described in any one of claims 1 to 6.

Images