A lane line recognition method, system, medium, device and information processing terminal

Abstract

The present invention belongs to the technical field of lane line recognition, and discloses a lane line recognition method, system, medium, device and information processing terminal, including: obtaining a binary image of the lane line and converting it into a grayscale image, and establishing a plane rectangular coordinate system; extracting the lane straight line vector and the coordinates of two endpoints on the fitted straight line in the coordinate system, and calculating the parameter values of the point-slope equation of the straight line; judging the slope, detecting the number of lane lines and the offset; calculating the intersection point of the straight line and the extracted straight line; traversing the array storing the x coordinates of the intersection points, and traversing the new array; obtaining the coordinates of the current lane line, and obtaining the offset of the vehicle in the current picture relative to the calculated lane line; obtaining the number of left and right lane lines by obtaining the lengths of the left and right lane line arrays. The lane line recognition method of the present invention is applicable to data extraction of images exported by neural network semantic segmentation, and has the characteristics of high extraction efficiency, fast speed, strong applicability, etc.

Description

Technical Field

[0001] The present invention belongs to the technical field of lane line recognition, and particularly relates to a lane line recognition method, system, medium, device and information processing terminal. Background Art

[0002] Currently, as driving assistance control for a vehicle, there are known various controls such as an adaptive cruise control system that selects a vehicle traveling in the same lane as the own vehicle as a preceding vehicle and tracks the selected preceding vehicle, and a lane keeping assistance that controls the driving of the vehicle so that the vehicle does not deviate from the left and right driving lane lines.

[0003] In such driving assistance control, a camera is mounted on the vehicle to photograph the front of the vehicle and identify the driving lane lines, and the driving of the vehicle is controlled using the identified driving lane lines. In related patent documents of existing lane line recognition, a method of calculating a clothoid parameter representing the curvature of a driving road based on the driving lane lines in an image captured by a camera and predicting the future behavior of a vehicle on the driving road using the calculated clothoid parameter is disclosed. However, the existing traditional lane line recognition method has a low recognition accuracy, and there is a problem of reduced controllability of vehicle driving control based on the shape of the driving lane lines. Therefore, there is an urgent need to design a new lane line recognition method and system.

[0004] Through the above analysis, the problems and defects of the existing technology are: the existing lane line recognition method has a low recognition accuracy, and there is a problem of reduced controllability of vehicle driving control based on the shape of the driving lane lines.

[0005] The difficulty of solving the above problems and defects is: traditional lane line recognition is greatly affected by actual environmental factors. It is difficult to reduce the degree of influence of the environment on lane line extraction and improve the accuracy of lane line extraction. On the other hand, it is also difficult to convert the extracted lane line image into control instructions that can be directly derived by the computer. In the process of identifying lane lines, it is found that in various complex environments, any algorithm cannot achieve high accuracy in identifying lane lines, so the function is clearly positioned as auxiliary driving, that is, after the confidence reaches a certain level, the data is collected and implemented. Lane line recognition initially tried the YOLO+LANNET method for recognition, in which YOLO recognizes road signs, pedestrians, and traffic lights. LANNET recognizes lane lines. The simulation runs well. After being transplanted to the robot, due to the limited resources of Jetson Nano, the two neural networks cannot run well, so we have to find another way. The recognition method was modified and YOLO+traditional visual recognition was tried. Although changing LANNET to traditional vision allows the car to run well. However, the effect of traditional visual recognition of lane lines is not very ideal and has low robustness. The car can only achieve better results when running in a venue with a constant light source, no reflections, and no dark spots.

[0006] The significance of solving the above problems and defects is: the present invention improves the YOLOP framework, without affecting the original three major visual tasks of traffic target detection, drivable area segmentation and lane line detection, and adds the lane line export image extraction function, and through a series of mathematical calculations, obtains the offset, the number of lane lines on the left and right sides, and the confidence. Compared with other traditional lane line recognition systems, it has higher accuracy. After processing the output results, by writing a ros program, the data is successfully reported to the move_base package in the ros navigation package. After priority processing, the lane keeping system is completed, which has strong robustness and can be applied to the Qingzhou robot. Summary of the invention

[0007] In response to the problems existing in the prior art, the present invention provides a lane line recognition method, system, medium, device and information processing terminal, and more particularly, relates to a lane line recognition method, system, medium, device and information processing terminal based on laser radar and deep learning ROS system.

[0008] The present invention is implemented as follows: a lane line recognition method, the lane line recognition method comprising the following steps:

[0009] Step 1: After obtaining the lane binary image derived from YOLOP, convert the image into a grayscale image. Converting to a grayscale image can greatly reduce the amount of calculation of the processing terminal;

[0010] Step 2: Take the top - left corner of the image as the origin, with the x - axis extending to the right from the origin and the y - axis extending downwards from the origin to establish a plane rectangular coordinate system, which can effectively restore the lane line equation.

[0011] Step 3: Extract the lane straight - line vectors that meet the conditions through Hough line fitting, extract the coordinates of two endpoints on the fitted line in the established coordinate system, and calculate the parameter values of the point - slope form equation of the line through the coordinates of the two endpoints, effectively finding the position of the lane line.

[0012] Step 4: After obtaining the line equation, judge the slope k. If the slope is between - 0.577 and 0.577, that is, between tan150° and tan30°, then it can be judged that this line should not be a lane line; determine a horizontal reference line, and detect the number of lane lines and the offset on this line through the reference line to improve the accuracy of the lane line.

[0013] Step 5: Calculate the intersection point of this line and the extracted line, and judge whether the x - coordinate of the calculated intersection point exceeds the range of the coordinate system. If it does not exceed, add the x - coordinate of this intersection point to the array list to reduce external interference.

[0014] Step 6: Traverse the array storing the x - coordinates of the intersection points, and perform a difference operation on adjacent data in the array. If the difference is less than a threshold, it represents a mutation point, and save the index and data of this mutation point, that is, the value of the coordinate x, to a new array respectively to improve the accuracy.

[0015] Step 7: Traverse the new array, divide the array into two parts by comparing the central value of the x - axis of the coordinate with the array values, and store them into two new arrays respectively. The new arrays are the x - coordinate values of the intersection points of the left and right lane lines and the reference line, accurately finding the parameter values.

[0016] Step 8: Take the last data of the left - lane - line array and the first data of the right - lane - line array respectively to obtain the coordinates of the two lane lines closest to the center of the screen, that is, the coordinates of the current lane lines, accurately finding the lane lines.

[0017] Step 9: Add the two obtained data and divide by 2 to calculate the x - coordinate value of the center point of the current lane line, and then subtract the central value of the x - axis of the coordinate to obtain the offset of the vehicle in the current screen relative to the calculated lane line; obtain the number of left and right lane lines by obtaining the lengths of the left and right lane - line arrays, accurately distinguishing the left and right lane lines.

[0018] Furthermore, in Step 3, the parameter values are a and b in y = ax + b; where, when the slope is 0, the two x or y coordinates of the extracted coordinate points are the same.

[0019] Further, in step three, the calculation formula is as follows:

[0020] k = (y1 - y2) / (x1 - x2), b = y2 - k * x2.

[0021] Further, in step four, the reference line is a straight line of y = a.

[0022] Another object of the present invention is to provide a lane line recognition system applying the above-mentioned lane line recognition method. The lane line recognition system includes:

[0023] An image conversion module, which is used to convert the lane line binary image exported by YOLOP into a grayscale image after obtaining it; establish a plane rectangular coordinate system with the upper left corner of the image as the origin, the right direction from the origin as the x-axis, and the downward direction from the origin as the y-axis;

[0024] A parameter value calculation module, which is used to extract the lane straight line vector meeting the conditions through Hough line fitting, extract the coordinates of two endpoints on the fitted straight line in the established coordinate system, and calculate the parameter values of the point-slope equation of the straight line through the coordinates of the two endpoints;

[0025] A slope judgment module, which is used to judge the slope k after obtaining the straight line equation. If the slope is between -0.577 and 0.577, that is, between tan150° and tan30°, it can be judged that this straight line should not be a lane line; determine a horizontal reference line, and detect the number of lane lines and the offset on this line through the reference line;

[0026] A straight line intersection calculation module, which is used to calculate the intersection of this straight line and the extracted straight line, and judge whether the x coordinate of the calculated intersection exceeds the range of the coordinate system. If it does not exceed, add the x coordinate of this intersection to the array list;

[0027] An array traversal module, which is used to traverse the array storing the x coordinates of the intersections, perform a difference operation on adjacent data of the array. If the difference is less than a threshold, it represents a mutation point, and save the index and data of this mutation point, that is, the value of the coordinate x, to a new array respectively;

[0028] A new array traversal module, which is used to traverse the new array, divide the array into two parts by comparing the center value of the x-axis of the coordinate with the array value, and store them into two new arrays respectively. The new arrays are the x coordinate values of the intersections of the left and right lane lines and the reference line;

[0029] A lane line coordinate acquisition module, which is used to respectively take the last data of the left lane line array and the first data of the right lane line array to obtain the coordinates of the two lane lines closest to the center of the picture, that is, the coordinates of the current lane lines;

[0030] The lane line number acquisition module is used to add the two acquired data and divide the sum by 2 to calculate the x - coordinate value of the center point of the current lane line, and then subtract the center value of the x - axis of the coordinate to obtain the offset of the vehicle in the current picture relative to the calculated lane line; the number of left and right lane lines is obtained by getting the lengths of the left and right lane line arrays.

[0031] Another object of the present invention is to provide a computer device, the computer device includes a memory and a processor, the memory stores a computer program, when the computer program is executed by the processor, the processor performs the following steps:

[0032] After obtaining the binary lane line image exported by YOLOP, convert the image into a grayscale image; establish a plane rectangular coordinate system with the upper - left corner of the image as the origin, the right direction from the origin as the x - axis, and the downward direction from the origin as the y - axis; extract the qualified lane line vector by Hough line fitting, extract the coordinates of two endpoints on the fitted line in the established coordinate system, and calculate the parameter values of the point - slope equation of the line through the two endpoint coordinates; after obtaining the line equation, judge the slope k, if the slope is between - 0.577 and 0.577, that is, between tan150° - tan30°, then it can be judged that this line should not be a lane line;

[0033] Determine a horizontal reference line, detect the number of lane lines and the offset on this line through the reference line; calculate the intersection point of this line and the extracted line, judge whether the x - coordinate of the calculated intersection point exceeds the coordinate system range, if not, add the x - coordinate of this intersection point to the array list; traverse the array storing the x - coordinates of the intersection points, perform a difference operation on adjacent data in the array, if the difference is less than a threshold, it represents a mutation point, and save the index and data of this mutation point, that is, the value of the x - coordinate, to a new array respectively; traverse the new array, divide the array into two parts by comparing the center value of the x - axis of the coordinate with the array values, and store them into two new arrays respectively, and the new arrays are the x - coordinate values of the intersection points of the left and right lane lines and the reference line;

[0034] Respectively take the last data of the left - lane - line array and the first data of the right - lane - line array to obtain the coordinates of the two lane lines closest to the center of the picture, that is, the coordinates of the current lane line; add the two obtained data and divide the sum by 2 to calculate the x - coordinate value of the center point of the current lane line, and then subtract the center value of the x - axis of the coordinate to obtain the offset of the vehicle in the current picture relative to the calculated lane line; the number of left and right lane lines is obtained by getting the lengths of the left and right lane line arrays.

[0035] Another object of the present invention is to provide a computer - readable storage medium, storing a computer program, when the computer program is executed by a processor, the processor performs the following steps:

[0036] After obtaining the binary image of the lane line exported by YOLOP, convert the image into a grayscale image; take the upper left corner of the image as the origin, with the x-axis extending to the right from the origin and the y-axis extending downward from the origin to establish a plane rectangular coordinate system; extract the lane line vector that meets the conditions through Hough line fitting, extract the coordinates of two endpoints on the fitted line in the established coordinate system, and calculate the parameter values of the point-slope equation of the line through the coordinates of the two endpoints; after obtaining the line equation, judge the slope k. If the slope is between -0.577 and 0.577, that is, between tan150° and tan30°, it can be judged that this line should not be a lane line;

[0037] Determine a horizontal reference line, and detect the number of lane lines and the offset on this line through the reference line; calculate the intersection point of this line and the extracted line, and judge whether the x coordinate of the calculated intersection point exceeds the range of the coordinate system. If it does not exceed, add the x coordinate of this intersection point to the array list; traverse the array storing the x coordinates of the intersection points, and perform a difference operation on adjacent data in the array. If the difference is less than a threshold, it represents a mutation point, and save the index and data of this mutation point, that is, the value of the coordinate x, to a new array respectively; traverse the new array, and divide the array into two parts by comparing the central value of the x-axis of the coordinate with the array value, and store them in two new arrays respectively. The new arrays are the x coordinate values of the intersection points of the left and right lane lines and the reference line;

[0038] Respectively take the last data of the left lane line array and the first data of the right lane line array to obtain the coordinates of the two lane lines closest to the center of the screen, that is, the current lane lines; add the two obtained data and divide by 2 to calculate the x coordinate value of the center point of the current lane line, and then subtract the central value of the x-axis of the coordinate to obtain the offset of the vehicle in the current screen relative to the calculated lane line; obtain the number of left and right lane lines by obtaining the lengths of the left and right lane line arrays.

[0039] Another object of the present invention is to provide a computer program product stored on a computer-readable medium, including a computer-readable program, which provides a user input interface to apply the lane line recognition system when executed on an electronic device.

[0040] Another object of the present invention is to provide a computer-readable storage medium storing instructions, which when run on a computer, enable the computer to apply the lane line recognition system.

[0041] Another object of the present invention is to provide an information data processing terminal, which is used to implement the lane line recognition system.

[0042] Combining all the above technical solutions, the advantages and positive effects of the present invention are as follows: The lane line recognition method provided by the present invention is applicable to data extraction from images derived from neural network semantic segmentation, and has the characteristics of high extraction efficiency, fast speed, and strong applicability. Traditional lane line recognition calculation methods have high requirements for computer hardware configuration, and the present invention is more suitable for application in small-scale unmanned driving. Through model improvement to conduct research on vision, each model has its own advantages and disadvantages. For example, compared with yolop, although the former can better complete the task, the model is too large and has high requirements for hardware, while the latter lightweight model is more suitable for the present invention. Therefore, when improving the model, attention should not only be paid to the effect, but also to whether the hardware meets the requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required for the embodiments of the present invention. Obviously, the following described drawings are only some embodiments of the present invention, and those of ordinary skill in the art can obtain other drawings based on these drawings without creative efforts.

[0044] Figure 1 It is a flowchart of the lane line recognition method provided by the embodiment of the present invention.

[0045] Figure 2 It is a structural block diagram of the lane line recognition system provided by the embodiment of the present invention;

[0046] In the figure: 1. Image conversion module; 2. Parameter value calculation module; 3. Slope judgment module; 4. Straight line intersection calculation module; 5. Array traversal module; 6. New array traversal module; 7. Lane line coordinate acquisition module; 8. Lane line quantity acquisition module.

[0047] Figure 3A and Figure 3B It is a binary map of the lane line exported by YOLOP provided by the embodiment of the present invention.

[0048] Figure 4 It is a schematic diagram of the simulation experiment scheme provided by the embodiment of the present invention.

[0049] Figure 5 It is a schematic diagram of local path planning provided by the embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0050] In order to make the objectives, technical solutions and advantages of the present invention clearer, the following further describes the present invention in detail with reference to the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0051] In view of the problems existing in the prior art, the present invention provides a lane line recognition method, system, medium, device and information processing terminal. The present invention will be described in detail below with reference to the accompanying drawings.

[0052] As Figure 1 shown, the lane line recognition method provided by the embodiment of the present invention includes the following steps:

[0053] S101, after obtaining the binary image of the lane line exported by YOLOP, convert the image into a grayscale image; establish a plane rectangular coordinate system with the upper left corner of the image as the origin, the right direction from the origin as the x-axis, and the downward direction from the origin as the y-axis;

[0054] S102, extract the lane straight line vectors that meet the conditions through Hough line fitting, extract the coordinates of two endpoints on the fitted straight line in the established coordinate system, and calculate the parameter values of the point-slope equation of the straight line through the coordinates of the two endpoints;

[0055] S103, after obtaining the straight line equation, judge the slope k. If the slope is between -0.577 and 0.577, that is, between tan150° - tan30°, it can be judged that this straight line should not be a lane line; determine a horizontal reference line, and detect the number of lane lines and the offset on this line through the reference line;

[0056] S104, calculate the intersection point of this straight line and the extracted straight line, and judge whether the x coordinate of the calculated intersection point exceeds the coordinate system range. If it does not exceed, add the x coordinate of this intersection point to the array list;

[0057] S105, traverse the array storing the x coordinates of the intersection points, and perform a difference operation on the adjacent data of the array. If the difference is less than a threshold, it represents a mutation point, and save the index and data of this mutation point, that is, the value of the coordinate x, to a new array respectively;

[0058] S106, traverse the new array, divide the array into two parts by comparing the center value of the x-axis of the coordinate with the array value, and store them in two new arrays respectively. The new arrays are the x coordinate values of the intersection points of the left and right lane lines and the reference line;

[0059] S107, respectively take the last data of the left lane line array and the first data of the right lane line array to obtain the coordinates of the two lane lines closest to the center of the screen, that is, the coordinates of the current lane line;

[0060] S108, add the two obtained data and divide by 2 to calculate the x coordinate value of the center point of the current lane line, and then subtract the center value of the x-axis of the coordinate to obtain the offset of the vehicle in the current screen relative to the calculated lane line; obtain the number of left and right lane lines by obtaining the lengths of the left and right lane line arrays.

[0061] As shown Figure 2 in the figure, the lane line recognition system provided by the embodiment of the present invention includes:

[0062] An image conversion module 1, which is used to convert the lane line binary image exported by YOLOP into a grayscale image after obtaining it; establish a plane rectangular coordinate system with the upper left corner of the image as the origin, the right direction from the origin as the x-axis, and the downward direction from the origin as the y-axis;

[0063] A parameter value calculation module 2, which is used to extract the lane straight line vectors that meet the conditions through Hough line fitting, extract the coordinates of two endpoints on the fitted straight line in the established coordinate system, and calculate the parameter values of the point-slope equation of the straight line through the coordinates of the two endpoints;

[0064] A slope judgment module 3, which is used to judge the slope k after obtaining the straight line equation. If the slope is between -0.577 and 0.577, that is, between tan150° - tan30°, it can be judged that the straight line should not be a lane line; determine a horizontal reference line, and detect the number of lane lines and the offset on this line through the reference line;

[0065] A straight line intersection calculation module 4, which is used to calculate the intersection of this straight line and the extracted straight line, and judge whether the x coordinate of the calculated intersection exceeds the range of the coordinate system. If it does not exceed, add the x coordinate of the intersection to the array list;

[0066] An array traversal module 5, which is used to traverse the array storing the x coordinates of the intersections, perform a difference operation on adjacent data in the array. If the difference is less than a threshold, it represents a mutation point, and save the index and data of the mutation point, that is, the value of the coordinate x, to a new array respectively;

[0067] A new array traversal module 6, which is used to traverse the new array, divide the array into two parts by comparing the center value of the x-axis of the coordinate with the array value, and store them into two new arrays respectively. The new arrays are the x coordinate values of the intersections of the left and right lane lines and the reference line;

[0068] A lane line coordinate acquisition module 7, which is used to respectively take the last data of the left lane line array and the first data of the right lane line array to obtain the coordinates of the two lane lines closest to the center of the picture, that is, the coordinates of the current lane lines;

[0069] A lane line number acquisition module 8, which is used to add the two obtained data and divide by 2 to calculate the x coordinate value of the center point of the current lane line, and then subtract the center value of the x-axis of the coordinate to obtain the offset of the vehicle in the current picture relative to the calculated lane line; obtain the number of left and right lane lines by obtaining the lengths of the left and right lane line arrays.

[0070] The technical solution of the present invention will be further described below in conjunction with specific embodiments.

[0071] Embodiment 1: Lane line data extraction method for neural network-derived images based on deep learning

[0072] The present invention provides a lane line recognition method for images exported by a YOLOP panoramic driving perception system. After obtaining the binary lane line image exported by YOLOP (see Figure 3A and Figure 3B ), the image is processed, and the specific method is as follows:

[0073] 1. Convert the image to a grayscale image.

[0074] 2. Establish a plane rectangular coordinate system with the upper left corner of the image as the origin, the right direction from the origin as the x-axis, and the downward direction from the origin as the y-axis.

[0075] 3. Extract the qualified lane line vector by Hough line fitting.

[0076] 4. Extract the coordinates of two endpoints on the fitted line in the established coordinate system.

[0077] 5. Calculate the parameter values of the point-slope equation of the line (a and b in y = ax + b) through these two endpoint coordinates. Special attention needs to be paid to the case where the slope is 0 (the two x or y coordinates of the extracted coordinate points are the same).

[0078] Calculation formula: k = (y1 - y2) / (x1 - x2), b = y2 - k * x2.

[0079] 6. After obtaining the line equation, judge the slope k. If the slope is between -0.577 and 0.577 (i.e., between tan(150°) and tan(30°)), it can be judged that this line should not be a lane line.

[0080] 7. Determine a horizontal reference line (i.e., a line of y = a), which is used to detect the number of lane lines and the offset on this line.

[0081] 8. Calculate the intersection point of this line and the extracted line, and judge whether the x coordinate of the calculated intersection point exceeds the coordinate system range. If it does not exceed, add the x coordinate of this intersection point to the array list.

[0082] 9. Then traverse the array storing the x coordinates of the intersection points, and perform a difference operation on the adjacent data of the array. If the difference is less than a threshold, it means reaching a mutation point, and save the index and data (the value of the coordinate x) of this mutation point to a new array respectively.

[0083] 10. Traverse the new array. By comparing the central value of the x-axis of the coordinates with the array values, divide the array into two parts and store them into two new arrays respectively. These two new arrays are the x-coordinate values of the intersections of the left and right lane lines and the reference line.

[0084] 11. To obtain the coordinates of the two lane lines closest to the center of the screen (i.e., the current lane lines), take the last data of the left lane line array and the first data of the right lane line array respectively.

[0085] 12. Add the two obtained data and divide by 2, that is, calculate the x-coordinate value of the center point of the current lane line, and then subtract the central value of the x-axis of the coordinates to obtain the offset of the vehicle in the current screen relative to the calculated lane line.

[0086] 13. By obtaining the lengths of the left and right lane line arrays, the number of the left and right lane lines can be obtained.

[0087] Embodiment 2

[0088] The embodiment of the present invention provides a lane line recognition method based on a binocular camera, a lidar, and a deep learning ROS system.

[0089] The present invention installs a development environment on the hardware basis of the Lightboat robot and drives the Lightboat robot to complete the autonomous driving task. Using Jetson nano as the industrial control computer, multiple experiments and developments such as ROS multi-machine communication, computer vision, laser SLAM, navigation positioning, and path planning are carried out on the Ubuntu18.04 system. Here, the vision based on the camera and the obstacle avoidance system based on the lidar are taken as the focus, and in cooperation with modules such as the motion system and mapping and positioning, the recognition of lane lines and various ground lane indication signs in the specified map is completed, as well as the recognition of pedestrians, zebra crossings, roadblocks, and traffic lights, and driving correctly according to the landmark indication and traffic light indication, automatically decelerating and giving way to pedestrians when encountering a zebra crossing and recognizing that there are pedestrians in the near distance directly ahead, and being able to change lanes to avoid obstacles without violating traffic rules, and finally successfully passing through the tunnel to complete all tasks.

[0090] 1. Technical solution (see Figure 4 )

[0091] 1.1 Mapping, positioning and path planning

[0092] Visual SLAM needs to complete the following steps:

[0093] 1. Read sensor data, such as single binocular cameras, depth cameras, lidars, etc.

[0094] 2. Visual odometry, whose function is to estimate the relative motion of adjacent two frames of pictures and the establishment of a local map.

[0095] 3. Back-end optimization, which receives the camera pose and loop detection information measured by the front-end.

[0096] 4. Loop detection, which is used to detect whether the robot has ever reached a previous position. If a loop is detected, it will transmit the information to the back-end for processing.

[0097] 5. Mapping, and finally mapping is performed according to the estimated trajectory.

[0098] Laser SLAM builds a map based on the point cloud information returned by the laser. The laser radar scans obstacles and is displayed in Rviz of the Ros system. The algorithm package is imported, and the map is obtained and saved with the help of a remote control. The A* algorithm and Dijkstra algorithm are used to implement global path planning, and the DWA algorithm is used to implement local path planning.

[0099] 1.2 Implementation in ROS

[0100] The ROS navigation package provides an algorithm, the AMCL (Adaptive Monte Carlo Localization) algorithm, which can track the current position of the robot through the depth data of the radar, combined with the odometry data, using a particle filter. However, here, because the road surrounding environment cannot be determined, it affects the core part of its algorithm, that is, the radar depth data is uncertain. Therefore, this algorithm is not suitable for this kind of task. Through reading and analyzing the source code implemented by AMCL for environmental perception - autonomous positioning, the matching part of its radar data was modified to directly use the odometry to publish coordinate transformation (that is, the relative position of the trolley), but the effect was not good. The reason is that there will be cumulative errors relying solely on the odometry, and it is not easy to solve through algorithms.

[0101] After consulting various materials, it was really difficult to find a method to localize the robot using existing sensors in this environment. Hardware was added, the T265 binocular camera. There is a highly optimized proprietary V-SLAM algorithm directly running on the Intel T265 binocular camera. Under the expected usage conditions, it can directly provide a closed-loop drift of less than 1%. A platform was made from wood, and then a framework was built on the platform to fix the T265 binocular camera on it. After configuring the TF transformation in the ROS system, the entire robot can successfully and accurately perceive its own position.

[0102] Semantic segmentation and instance segmentation based on deep learning can achieve lane line detection. A conda virtual environment is constructed, labeled with the labelme tool, and a dataset for training is made using a.bat script. To test the effect of the segmentation task, two annotation methods, polygon and straight line, are used here. After comparison, it is found that the polygon method can complete the task better, so the dataset with polygon annotation is adopted. At the same time, the scheme uses two models to detect lane lines. a. The U-Net network is one of the algorithms for semantic segmentation using a fully convolutional network. It was initially used in the field of medical imaging, but due to its simplicity, efficiency, understandability, and ease of construction, it has been used in other fields. It is precisely these advantages that this invention uses it as one of the models for lane line detection. Putting the previously made dataset into the model for training to obtain a weight file can be used for prediction. Of course, although the U-Net network is simple and has good effects, its disadvantages are also obvious. It cannot classify the detection of uncertain multiple types and quantities of lane lines. b. Using the Lanenet model, the main network part of Lanenet has two branches. One branch predicts the mask, and the other branch assigns the id of the lane to which each lane pixel belongs. Therefore, it can solve the problems that the U-Net cannot solve. The disadvantage is that the network is relatively complex, but the training and prediction steps are similar.

[0103] Lane line recognition is a very important part of this invention. However, during the process of the car recognizing lane lines, it is found that in various complex environmental situations, no algorithm can achieve a very high accuracy in lane line recognition. Therefore, the function is clearly positioned as assisting driving, that is, after the confidence level reaches a certain degree, the data is collected and realized. Initially, the method of yolo + lannet was tried for lane line recognition, where yolo recognizes road signs, pedestrians, and traffic lights, and lannet recognizes lane lines. The simulation runs with good results. After being transplanted to the robot, due to the limited resources of the jetson nano, the two neural networks cannot run well, so a new approach has to be taken. The recognition method is modified and the attempt is made to use yolo + traditional vision recognition. Although changing lannet to traditional vision enables the car to run well, the effect of traditional vision in recognizing lane lines is not very ideal, with low robustness. The car can only achieve a better effect when running on a site with constant light, no reflection, and no dark spots.

[0104] Finally, through literature research, the YOLOP framework was improved. This framework is called the Panoramic Driving Perception Network, which can simultaneously perform three visual tasks: traffic target detection, drivable area segmentation, and lane line detection. The lane lines were extracted and located, and finally the output offset and the number of left and right lane lines were obtained. After processing the output results, by writing a ROS program, the data was successfully reported to the move_base package in the ROS navigation package. After priority processing, the lane keeping system was completed.

[0105] Object detection based on deep learning is used to detect road signs, obstacles, crosswalks, etc. Similarly, the first step is to create a dataset. The dataset is obtained by manually annotating using labelimg in a virtual environment. There are many object detection algorithms, including SSD, YOLO series, Fast-RCNN, etc. In this invention, YOLOv5 is used to achieve the recognition of road signs. The dataset is put into YOLOv5 for training and prediction to complete the road sign recognition. Since the shooting angle of the landmarks is very small, the angles of the dataset images are almost the same, and the robustness of the model is poor. Therefore, image stretching transformation can be used to achieve the effect that the road signs are photographed at various angles visually. Of course, more datasets can also be found for training.

[0106] As Figure 5 shown, the obstacle avoidance algorithm used in this invention is the local path planning in the ROS navigation package, mainly using the dynamic window method. This algorithm can receive external obstacle information in real time to achieve obstacle avoidance. For the local path planning - obstacle avoidance algorithm, when a stationary obstacle is detected ahead, the vehicle will perform path planning to avoid the obstacle. When a pedestrian is detected ahead through YOLOP, through the distance measurement of the radar, the vehicle will stop within a safe range.

[0107] 2. This invention realizes the method of lane line recognition and unmanned driving technology, that is, navigation + computer vision. During the navigation process, the vehicle not only needs to estimate its own position but also needs to plan the movement route. This invention has high requirements for computer hardware and requires high computing power. Therefore, this invention is more suitable for application in small-scene unmanned driving. Through model improvement to study vision, each model has its own advantages and disadvantages. For example, compared with YOLOP, although YOLO + Lannet can complete the task better, the model is too large and has high requirements for hardware, while the latter lightweight model is more suitable for this invention. Therefore, when improving the model, not only the effect should be focused on, but also whether the hardware meets the requirements should be considered, so as to complete visual detection.

[0108] The technical effects of the present invention will be described in detail below in combination with experiments.

[0109] Table 1 Comparison of the extraction accuracy of several lane line extraction neural networks

[0110] Network Accuracy (%) ENet 34.12 SCNN 35.79 ENET-SAD 36.56 YOLOP 70.50

[0111] In a complex environment, compare day and night; high and low traffic volume; dim and bright roadside street lights. Under different conditions, compare the lane line recognition effects of different models. The results prove that the present invention can accurately detect and mark lane lines.

[0112] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in whole or in part in the form of a computer program product, the computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, the processes or functions described in the embodiments of the present invention are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wire (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer, or a data storage device such as a server or data center that includes one or more integrated available media. The available medium can be a magnetic medium (such as a floppy disk, hard disk, magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid-state drive (SSD)).

[0113] The above is only the specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention, any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention shall be covered by the protection scope of the present invention.

Claims

1. A lane line recognition method, characterized in that, The lane line recognition method includes the following steps: Step 1, after obtaining the binary image of the lane line exported by YOLOP, convert the binary image of the lane line into a grayscale image; Step 2, take the upper left corner of the image as the origin, with the right direction from the origin as the x-axis and the downward direction from the origin as the y-axis to establish a plane rectangular coordinate system; Step 3, extract the qualified lane line vector by Hough line fitting, extract the coordinates of two endpoints on the fitted line in the established coordinate system, and calculate the parameter values of the point-slope equation of the line through the coordinates of the two endpoints; Step 4, after obtaining the line equation, judge the slope a. If the slope is between -0.577 and 0.577, it is judged that the lane line with a qualified slope should not be a lane line; determine a horizontal reference line, and detect the number of lane lines and the offset on the reference line through the reference line; Step 5, calculate the intersection points of the lane line with a qualified slope and the qualified lane line extracted by Hough line fitting, and judge whether the x coordinate of the calculated intersection point exceeds the coordinate system range. If it does not exceed, add the x coordinate of the intersection point to the array list; Step 6, traverse the array storing the x coordinates of the intersection points, perform a difference operation on adjacent data in the array. If the difference is less than a threshold, it represents a mutation point, and save the index of the mutation point and the value of the x coordinate of the mutation point into a new array respectively; Step 7, traverse the new array, compare the center value of the x-axis coordinate with the array value. The array values less than the center value of the x-axis are stored in an array, representing the x coordinate values of the intersection points of the left lane line and the reference line, and the array values greater than the center value of the x-axis are stored in another array, representing the x coordinate values of the intersection points of the right lane line and the reference line; Step 8, respectively take the last data of the left lane line array and the first data of the right lane line array to obtain the coordinates of the two lane lines and the current lane line closest to the center of the picture; Step 9, add the two obtained data and divide by 2 to calculate the x coordinate value of the center point of the current lane line, and then subtract the center value of the x-axis coordinate to obtain the offset of the vehicle in the current picture relative to the calculated lane line; obtain the number of left and right lane lines by obtaining the lengths of the left and right lane line arrays.

2. The lane line recognition method according to claim 1, characterized in that, In Step 3, the parameter values are a and b in y = ax + b; where, when the slope is 0, the two x or y coordinates of the extracted coordinate points are the same.

3. The lane line recognition method according to claim 1, characterized in that, In Step 3, the calculation formula is: a = (y1 - y2) / (x1 - x2), b = y2 - a * x2.

4. The lane line recognition method according to claim 1, characterized in that, In Step 4, the reference line is a straight line of y = a.

5. A lane line recognition system, characterized in that, The lane line recognition system includes: An image conversion module, which is used to convert the binary image of the lane line into a grayscale image after obtaining the binary image of the lane line exported by YOLOP; take the upper left corner of the image as the origin, with the right direction from the origin as the x-axis and the downward direction from the origin as the y-axis to establish a plane rectangular coordinate system; A parameter value calculation module, which is used to extract the qualified lane line vector by Hough line fitting, extract the coordinates of two endpoints on the fitted line in the established coordinate system, and calculate the parameter values of the point-slope equation of the line through the coordinates of the two endpoints; The slope judgment module is used to judge the slope a after obtaining the straight-line equation. If the slope is between -0.577 and 0.577, it is judged that the lane line straight line with a qualified slope should not be a lane line; determine a horizontal reference line, and detect the number of lane lines and the offset on this line through the reference line; The straight-line intersection calculation module is used to calculate the intersection points of the lane line straight lines with qualified slopes and the lane line straight lines with qualified slopes extracted by Hough straight-line fitting, and judge whether the x coordinates of the calculated intersection points exceed the coordinate system range. If not, add the x coordinates of the intersection points to the array list; The array traversal module is used to traverse the array storing the x coordinates of the intersection points, and perform a difference operation on adjacent data in the array. If the difference is less than a threshold, it represents a mutation point, and save the index and data of the mutation point, that is, the value of the coordinate x, to a new array respectively; The new array traversal module is used to traverse the new array. By comparing the center value of the x axis of the coordinate with the array value, the array is divided into two parts and stored in two new arrays respectively. The new arrays are the x coordinate values of the intersection points of the left and right lane lines and the reference line; The lane line coordinate acquisition module is used to respectively take the last data of the left lane line array and the first data of the right lane line array to obtain the coordinates of the two lane lines closest to the center of the screen, that is, the coordinates of the current lane lines; The lane line number acquisition module is used to add the two obtained data and divide by 2 to calculate the x coordinate value of the center point of the current lane line, and then subtract the center value of the x axis of the coordinate, that is, obtain the offset of the vehicle in the current screen relative to the calculated lane line; obtain the number of left and right lane lines by obtaining the lengths of the left and right lane line arrays.

6. A computer device, characterized in that, The computer device includes a memory and a processor. When the computer program stored in the memory is executed by the processor, the processor executes the lane line recognition method according to any one of claims 1-4, including the following steps: After obtaining the lane line binary image exported by YOLOP, convert the lane line binary image into a grayscale image; establish a plane rectangular coordinate system with the upper left corner of the image as the origin, the right direction from the origin as the x axis, and the downward direction from the origin as the y axis; extract the qualified lane straight line vectors by Hough straight-line fitting, extract the coordinates of the two end points on the fitted straight line in the established coordinate system, and calculate the parameter values of the point-slope equation of the straight line through the two end point coordinates; after obtaining the straight-line equation, judge the slope a. If the slope is between -0.577 and 0.577, it is judged that the lane line straight line with a qualified slope should not be a lane line; Determine a horizontal reference line, and detect the number of lane lines and the offset on this line through the reference line; calculate the intersection points of the lane line straight lines with qualified slopes and the qualified lane line straight lines extracted by Hough line fitting, and judge whether the x coordinate of the calculated intersection point exceeds the coordinate system range. If it does not exceed, add the x coordinate of the intersection point to the array list; traverse the array storing the x coordinates of the intersection points, and perform a difference operation on adjacent data in the array. If the difference is less than a threshold, it represents a mutation point, and save the index and data of the mutation point, that is, the value of the coordinate x, to a new array respectively; traverse the new array, and divide the array into two parts by comparing the center value of the x axis of the coordinate with the array value, and store them in two new arrays respectively. The new arrays are the x coordinate values of the intersection points of the left and right lane lines and the reference line. Respectively take the last data of the left lane line array and the first data of the right lane line array to obtain the coordinates of the two lane lines closest to the center of the screen, that is, the coordinates of the current lane lines; add the two obtained data and divide by 2 to calculate the x coordinate value of the center point of the current lane line, and then subtract the center value of the x axis of the coordinate, that is, obtain the offset of the vehicle in the current screen relative to the calculated lane line; obtain the number of left and right lane lines by obtaining the lengths of the left and right lane line arrays.

7. A computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the processor executes the lane line recognition method according to any one of claims 1-4, including the following steps: After obtaining the lane line binary image exported by YOLOP, convert the lane line binary image into a grayscale image; establish a plane rectangular coordinate system with the upper left corner of the image as the origin, the right direction from the origin as the x axis, and the downward direction from the origin as the y axis; extract the qualified lane straight line vectors by Hough line fitting, extract the coordinates of two end points on the fitted straight line in the established coordinate system, and calculate the parameter values of the point-slope equation of the straight line through the coordinates of the two end points; after obtaining the straight line equation, judge the slope a. If the slope is between -0.577 and 0.577, it is judged that the lane line straight line with a qualified slope should not be a lane line. Determine a horizontal reference line, and detect the number of lane lines and the offset on this line through the reference line; calculate the intersection points of the lane line straight lines with qualified slopes and the qualified lane line straight lines extracted by Hough line fitting, and judge whether the x coordinate of the calculated intersection point exceeds the coordinate system range. If it does not exceed, add the x coordinate of the intersection point to the array list; traverse the array storing the x coordinates of the intersection points, and perform a difference operation on adjacent data in the array. If the difference is less than a threshold, it represents a mutation point, and save the index and data of the mutation point, that is, the value of the coordinate x, to a new array respectively; traverse the new array, and divide the array into two parts by comparing the center value of the x axis of the coordinate with the array value, and store them in two new arrays respectively. The new arrays are the x coordinate values of the intersection points of the left and right lane lines and the reference line. Respectively take the last data of the left lane line array and the first data of the right lane line array to obtain the two lane lines closest to the center of the screen, that is, the coordinates of the current lane line; add the two obtained data and divide by 2 to calculate the x coordinate value of the center point of the current lane line, and then subtract the center value of the coordinate x axis to obtain the offset of the vehicle in the current screen relative to the calculated lane line; obtain the number of left and right lane lines by obtaining the lengths of the left and right lane line arrays.

Images