Vehicle lane keeping system and method based on machine self-learning

Abstract

The present invention relates to a vehicle lane keeping system based on machine self-learning, comprising a sensor perception module and a self-learning control module. The sensor perception module is used to sense the vehicle's environmental state and transmit this information to the self-learning control module. The self-learning control module learns lane keeping strategies through continuous interaction with the environment. The present invention utilizes a neural network for end-to-end control, achieving more abstract control and enabling effective control in various states. Reinforcement learning is also employed to impart a degree of self-learning capability to the algorithm. Image processing techniques are incorporated into the shallow layers of the corresponding network to accelerate neural network learning.

Description

Technical Field

[0001] The present invention belongs to the technical field of autonomous vehicle lane keeping, and specifically relates to a vehicle lane keeping system and method based on machine self-learning. Background Art

[0002] With the advancement and development of technology, artificial intelligence (AI), such as Siri, voice input, intelligent robots, and autonomous driving, will gradually permeate our lives. The importance of artificial intelligence in autonomous driving is also attracting considerable research interest. The development of autonomous driving technology can significantly improve people's lives while also preventing traffic congestion and even accidents caused by human error. As of the end of 2018, approximately 1.35 million people died in road traffic accidents worldwide each year. Autonomous vehicles, based on modern technology, have enormous potential in terms of safety and reliability. Therefore, the development of autonomous vehicles is crucial for reducing traffic accidents and protecting people's lives.

[0003] The traditional lane keeping system architecture consists of recognition, analysis, decision-making, and control execution systems. The recognition module processes camera information, while the analysis module processes sensor information to determine the vehicle's current position relative to the lane line, direction of deviation, and speed. When the vehicle approaches the identified boundary line and is about to leave the lane, the decision-making system uses a fixed strategy to control the vehicle to maintain lane control. However, this approach may lead to the following problems:

[0004] (1) The control strategy is relatively rigid and not flexible enough;

[0005] (2) Roads must express lane lines according to certain standards;

[0006] (3) The vehicle does not have sufficient adaptive capabilities.

[0007] Currently, traditional lane detection methods use sensors to collect lane image information, perform preliminary processing on the input information, and then detect lane positions in real time. Research on lane detection technology began relatively early abroad. As early as the 1990s, lane detection has become a component of driver assistance systems. The GOLD system, proposed by researchers at the University of Parma in Italy, has achieved lane and obstacle detection. It uses inverse perspective to transform lane lines into parallel lines and employs template matching to detect and determine lane positions. Feng You et al. proposed using image processing methods to process digital images captured by CCD cameras for nighttime lane detection. They used a multi-directional search method to remove noise from lane boundary images and employed an adaptive Hough transform to detect lane lines. This algorithm demonstrates greater reliability and robustness for nighttime lane detection. Researchers at Duke University proposed the LaneNet method, a deep neural network-based lane detection method that divides lane detection into two stages: lane edge extraction and lane localization. This algorithm significantly reduces false alarm rates and achieves excellent detection results.

[0008] In lane keeping assist systems (LKAS), active lane keeping control is the most effective method for risk avoidance. A well-designed lane keeping lateral control algorithm allows the system to quickly and effectively control the vehicle and eliminate lane departures. Based on different design approaches, lane keeping lateral control algorithms can be broadly categorized into two types: those based on driver models and those based on automatic control theory.

[0009] A search found that CN113844448A discloses a lane keeping method based on deep reinforcement learning. This method first collects environmental state data and vehicle state data, pre-processes them, and then passes them into the network model. The lane keeping driving task is learned by reinforcement learning based on the interactive data. During this period, the network is continuously updated until the model converges. Then, the optimal strategy output control for task execution is performed based on the converged network. This method does not meet the conditions for deep self-learning. Summary of the Invention

[0010] The purpose of the present invention is to address the defects of the existing technology and propose a vehicle lane keeping system and method based on machine self-learning, which can realize lane keeping during the automatic driving of the vehicle.

[0011] In order to achieve the above objectives, the present invention provides a vehicle lane keeping system based on machine self-learning, comprising a sensor perception module and a self-learning control module, wherein the sensor perception module is used to perceive the environmental state of the vehicle and send the perceived environmental state information to the self-learning control module;

[0012] The self-learning control module self-learns lane keeping strategies through continuous interaction with the environment.

[0013] The neural network in the present invention adopts the transfer learning method in deep learning when processing the environmental state to accelerate the self-learning process of the vehicle intelligent body.

[0014] The present invention discloses a vehicle lane keeping method based on machine self-learning. The self-learning control method is to learn the relevant lane keeping strategy by continuously interacting with the environment, and specifically includes the following steps:

[0015] Step 1: Initialize the vehicle state so that it is randomly placed in the lane keeping task environment; then proceed to step 2;

[0016] Step 2: The sensor perception module converts the data collected by the sensor into a corresponding matrix to describe the vehicle status through perception of the environment; then proceed to step 3;

[0017] Step 3: The self-learning control module calculates the corresponding vehicle state, updates its own control strategy, calculates the control amount that the vehicle should take, and then sends the control amount to the controlled vehicle; then proceeds to step 4;

[0018] Step 4: The vehicle takes the corresponding control amount to interact with the environment. After the interaction is completed, it reaches the next state and calculates the reward given by the environment during this process. Then go to step 5.

[0019] Step 5: Determine whether the vehicle has learned the corresponding mature strategy. If the vehicle has learned the corresponding mature strategy, the self-learning process ends; otherwise, proceed to step 6.

[0020] Step 6: Determine whether the vehicle's state has reached the task completion state. If the vehicle's state is the task completion state, package the vehicle state, control amount, next state, reward, and completion flag into a five-tuple, put it into the algorithm cache, and then return to step 1; otherwise, return to step 3.

[0021] The present invention further adopts the following technical solution:

[0022] In step 3, during the control strategy update process, the self-learning control module has a total of four neural networks, namely the value network, the target value network, the action network and the target action network. The resnet34 feature extraction network is migrated to the shallow network of the four neural networks (the resnet34 network removes the last linear layer).

[0023] The specific steps of step 3 are as follows:

[0024] Step 3-1: Extract some five-tuples (including vehicle state, control amount, next state, reward, and completion flag) from the algorithm cache; then go to step 3-2;

[0025] Step 3-2: The target action network selects the control amount to be taken in the next state through the extracted next state; go to step 3-3;

[0026] Step 3-3, the target value network calculates the value of the control quantity adopted in the next state; and proceeds to step 3-4;

[0027] Step 3-4: The value network calculates the current value based on the current state and the action taken under the current action. Then, it updates its network parameters based on the value of the next state control amount (i.e., the sampled reward) calculated by the target value network. Go to step 3-5.

[0028] Step 3-5: The action network updates its own network parameters based on the actions taken and the evaluation of the actions by the value network; then proceed to step 3-6;

[0029] Step 3-6: At regular intervals, shallowly copy the network parameters of the action network and the value network to the value network and the target action network.

[0030] In step 6, the criterion for determining whether the vehicle state has reached the task completion state is: when the vehicle body posture is parallel to the lane line center and the vehicle is in the lane line center, the task is completed.

[0031] The advantages of the present invention are as follows:

[0032] (1) Using neural networks to process end-to-end control, the control is more abstract and can effectively perform relevant control in various states;

[0033] (2) Use reinforcement learning to give the algorithm a certain degree of self-learning ability;

[0034] (3) Add image processing technology to the shallow network of the corresponding network to make the neural network learn faster. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The present invention will be further described below with reference to the accompanying drawings.

[0036] Figure 1 This is a system framework diagram of the present invention.

[0037] Figure 2 Flow chart of the method of the present invention. DETAILED DESCRIPTION Example 1

[0038] like Figure 1As shown, a vehicle lane keeping system based on machine learning includes a sensor perception module and a self-learning control module. The sensor perception module senses the vehicle's environment and transmits this information to the self-learning control module. The self-learning control module learns lane keeping strategies through continuous interaction with the environment. The sensor perception module includes an onboard camera, an IMU, a vehicle acoustic velocity sensor, a GPS sensor system, and a lane detection sensor. The sensor perception module senses the vehicle's state. The self-learning control module learns the vehicle's control strategy through continuous interaction with the vehicle's environment. Based on the perception module's understanding of the environment, the vehicle's state is defined as the camera information in front of the vehicle (a 224x224x3 image), the vehicle's IMU information (a 1x6 vector representing [linear acceleration x, linear acceleration y, linear acceleration z, angular velocity x, angular velocity y, angular velocity z]), the vehicle's own speed, and the vehicle's desired speed. The self-learning control module outputs a three-dimensional vector, consisting of [throttle, steering wheel angle] (when the throttle is less than 0, it represents the degree of braking). The vehicle's self-learning control module eventually learns the relevant control strategies for vehicle lane keeping during continuous interaction with the environment.

[0039] like Figure 2 As shown in FIG, a self-learning control method for a vehicle lane keeping system based on machine self-learning is described. The self-learning control method learns the lane keeping strategy by continuously interacting with the environment. The specific steps are as follows:

[0040] Step 1: Initialize the vehicle state so that it is randomly in the lane keeping task environment; then go to step 2.

[0041] Step 2: The sensor perception module uses its environmental awareness to convert sensor data into a matrix to describe the vehicle's state. Specifically, the module uses the vehicle's onboard camera to obtain a photo of the road ahead and converts it into a 224x224x3 image. It then uses the IMU sensor to obtain the vehicle's three-axis acceleration and three-axis angular acceleration information, while the speed sensor obtains the current speed information. These information are used to describe the vehicle's state. The process then proceeds to Step 3.

[0042] Step 3: The action network in the self-learning control module calculates the action that the vehicle should take, that is, the control amount that the vehicle should take, by calculating the corresponding vehicle state, and sends the control amount to the controlled vehicle; then proceed to step 4.

[0043] Step 4: The vehicle executes the control value calculated in Step 3 to interact with the environment. After the interaction is completed, it reaches the next state. The reward obtained by the vehicle in this process is calculated by the distance between the vehicle posture in this state and the lane line posture, and then proceeds to Step 5.

[0044] Step 5: Determine whether the vehicle has learned the corresponding mature strategy, that is, determine whether the control strategy learned by the vehicle has converged. If the vehicle's control strategy can always enable the vehicle to complete the lane keeping task during multiple control processes, it means that the vehicle has learned the corresponding mature strategy and the self-learning process is complete. Otherwise, proceed to Step 6.

[0045] Step 6: Pack the vehicle state, control amount, next state, reward, and completion flag into experience data and put it into the algorithm's cache. At the same time, the algorithm extracts the cached experience data and optimizes its own network based on the rewards and actions obtained by the vehicle in the experience, so that its own strategy is optimized in the direction of increasing the total reward; then go to step 7.

[0046] Step 7: When the vehicle's body posture is parallel to the center of the lane line and the vehicle is at the center of the lane line, the task is completed and the process returns to step 1; otherwise, the process returns to step 3.

[0047] When the vehicle updates its control strategy, the self-learning control module has four neural networks: the value network, the target value network, the action network, and the target action network. The ResNet34 feature extraction network (the last linear layer of the ResNet34 network is removed) is migrated to the shallow layers of these four neural networks. The specific steps are as follows:

[0048] Step 3-1: Extract some five-tuples (vehicle state, control amount, next state, reward, completion flag) from the algorithm cache; go to step 3-2;

[0049] Step 3-2: The target action network extracts the next state and calculates the vehicle action that should be taken in the next state through the neural network. That is, the control amount of the vehicle is transferred to step 3-3;

[0050] Step 3-3: The target value network calculates the action value of the next state through the neural network based on the next state and the vehicle action obtained in step 3-2, and then proceeds to step 3-4;

[0051] Step 3-4: The value network calculates the vehicle's action value through the neural network using the extracted vehicle state and the control vehicle state. It then updates its own network parameters based on the action value of the next state calculated in step 3-2 and the reward data in the experience, so that the vehicle's action value is equal to the sum of the action value of the next state and the reward. Go to step 3-5.

[0052] Step 3-5: The action network updates its own network parameters based on the action taken and the action value obtained in step 3-4, and updates the output of the action network in the direction of obtaining a higher action value; then proceed to step 3-6;

[0053] Step 3-6: At regular intervals, shallowly copy the network parameters of the action network and the value network to the value network and the target action network.

[0054] In addition to the above embodiments, the present invention may also have other implementations. Any technical solution formed by equivalent replacement or equivalent transformation falls within the scope of protection required by the present invention.

Claims

1. A vehicle lane keeping method based on machine self-learning, characterized in that: The system involved in the method includes a sensor perception module and a self-learning control module, wherein the sensor perception module is used to perceive the environmental state of the vehicle and send the perceived environmental state information to the self-learning control module; The self-learning control module self-learns lane keeping strategies through continuous interaction with the environment; The method comprises the following steps: Step 1: Initialize the vehicle state so that it is randomly placed in the lane keeping task environment; then proceed to step 2; Step 2: The sensor perception module converts the data collected by the sensors into a corresponding matrix to describe the vehicle status through environmental perception. The sensor perception module obtains a photo of the road in front of the vehicle through the onboard camera and converts it into a 224x224x3 image. The IMU sensor then obtains the vehicle's three-axis acceleration and three-axis angular acceleration information. At the same time, the speed sensor obtains the current speed information. The above information is used to describe the vehicle status. Go to step 3; Step 3: The self-learning control module updates its control strategy by calculating the corresponding vehicle state, calculates the control amount that the vehicle should take, and then sends the control amount to the controlled vehicle. During the control strategy update process, the self-learning control module has a total of four neural networks: the value network, the target value network, the action network, and the target action network. The ResNet34 feature extraction network is migrated to the shallow network of these four neural networks. The specific steps are as follows: Step 3-1: Extract some quintuples from the algorithm cache; go to step 3-2; Step 3-2: The target action network selects the control amount to be taken in the next state through the extracted next state; go to step 3-3; Step 3-3, the target value network calculates the value of the control quantity adopted in the next state; and proceeds to step 3-4; Step 3-4: The value network calculates the current value based on the current state and the action taken under the current action, and then updates its network parameters based on the value of the next state control quantity calculated by the target value network; go to step 3-5; Step 3-5: The action network updates its own network parameters based on the actions taken and the evaluation of the actions by the value network; then proceed to step 3-6; Step 3-6: At regular intervals, shallowly copy the network parameters of the action network and the value network to the value network and the target action network; Go to step 4; Step 4: The vehicle takes the corresponding control amount to interact with the environment. After the interaction is completed, it reaches the next state and calculates the reward given by the environment during this process. Then go to step 5. Step 5: Determine whether the vehicle has learned the corresponding mature strategy, that is, determine whether the control strategy learned by the vehicle has converged. If the vehicle's control strategy can always complete the lane keeping task during multiple control processes, it means that the vehicle has learned the corresponding mature strategy. If the vehicle has learned the corresponding mature strategy, the self-learning process ends; otherwise, proceed to step 6. Step 6: Determine whether the vehicle's state has reached the task completion state. If the vehicle's state is the task completion state, the vehicle state, control amount, next state, reward, and completion flag are packaged into a five-tuple and placed in the algorithm cache. The algorithm extracts the cached experience data and optimizes its own network based on the rewards and actions obtained by the vehicle in the experience, so that its own strategy is optimized in the direction of increasing the total reward, and then returns to step 1; otherwise, returns to step 3.

Images