Lane line detection system based on FPN under monitoring

Abstract

The invention relates to the technical field of lane line detection, and the method comprises the steps: preparing data, and preprocessing a monitoring video; multi-level features are extracted through FPN; detecting a lane line starting point according to the feature and a conditional convolution kernel; describing lane lines according to a line formula; detecting a lane line; performing perspective inverse transformation according to the coordinates of the lane line, obtaining the pixel coordinates of the lane line after the lane line is detected, and performing perspective inverse transformation on the picture on the basis of the pixel coordinates; a mapping relation is constructed by combining pixel coordinates and actual lane line information, and vehicle detection and tracking are carried out at the same time; a vehicle speed is estimated. Accurate data are provided for automatic driving and the detection accuracy is improved by processing a monitoring video, extracting and unifying picture resolution, extracting multilayer features by using FPN, obtaining pixel coordinates after detection, performing perspective inverse transformation, converting a lane line into a top view, forming mapping in combination with actual information, and estimating the vehicle speed.

Description

Technical Field

[0001] The present invention relates to the technical field of lane line detection, and more specifically, to a lane line detection system based on FPN under monitoring. Background Art

[0002] In the past two years, autonomous driving has developed particularly rapidly. With the perfect combination of "Internet +" and mobile terminals, the autonomous driving system is mainly composed of a perception system and a decision system. The perception system is responsible for positioning, obstacle detection, road mapping, etc., while the decision system handles tasks such as path planning and behavior selection. A crucial part of this is the planning of drivable paths, and the prerequisite for this step is to detect the lane markings on the road surface. At the same time, lane line detection is also crucial for identifying illegal driving behaviors.

[0003] Lane detection algorithms are mainly divided into two categories: traditional vision-based algorithms and deep learning-based algorithms. Traditional algorithms are further divided into feature-based and model-based algorithms. Feature-based algorithms detect lane lines by extracting information such as the color, texture, direction, and shape of lane lines. Model-based algorithms use mathematical modeling, such as using a curve model to describe the shape of lane lines, and modeling lane lines as straight lines or high-order curves.

[0004] Due to the long and slender shape of lane lines, and the lack of visual information such as complex lane line changes, occlusion problems, lighting effects, and semantic ambiguity in real scenes, it is difficult to handle the above challenging scenes when instance-level lane line recognition is required. In addition, lane line detection requires effective identification of lane lines in scenes under extreme conditions of severe occlusion, damage, or very dim light, which requires contextual information and global information based on images, which will bring a huge amount of calculations and is difficult to apply in practice. Therefore, it is necessary to design a lane line detection system based on FPN under surveillance. Summary of the invention

[0005] In order to overcome the shortcomings of the prior art, the present invention provides a lane line detection system based on FPN under monitoring, which has the advantages of applying the FPN structure to lane line detection, improving the accuracy and robustness of detection, and estimating the vehicle speed through inverse perspective transform.

[0006] To achieve the above object, the present invention provides the following technical solution: a lane line detection system based on FPN under monitoring, comprising:

[0007] Data preparation: pre-process the surveillance video, adjust the image size to 1080P, and also mark the lane lines during the training phase;

[0008] FPN extracts multi-level features and uses FPN to extract multi-level features from the input image;

[0009] The lane line starting point is detected based on the features combined with the conditional convolution kernel, and the features at the high-level semantic level are used to detect the lane line starting point;

[0010] The lane line is described according to the row formula, and the low-level semantic features are used to describe the details of the lane line shape;

[0011] Detect lane lines;

[0012] Perform inverse perspective transformation based on the lane line coordinates. After detecting the lane line, obtain the pixel coordinates of the lane line, and perform inverse perspective transformation on the image based on this.

[0013] Combine pixel coordinates and actual lane line information to build a mapping relationship, and perform vehicle detection and tracking at the same time;

[0014] Estimate vehicle speed.

[0015] As a preferred technical solution of the present invention, the data preparation includes a training set and a test set, both of which are self-made data sets. The video source is video surveillance of a highway. The surveillance video is extracted by frame-by-frame extraction to obtain pictures. After extracting the frames, the pictures are first resized to uniformly adjust the resolution to 1920×1080, and then the blurred images are manually removed.

[0016] As a preferred technical solution of the present invention, the detected lane lines are detected by a lane line detection model based on FPN and conditional convolution kernel, and the lane lines are marked from left to right using Labelme.

[0017] As a preferred technical solution of the present invention, the lane line detection model based on FPN and conditional convolution kernel adopts Resnet as the backbone network, and introduces FPN to extract multi-semantic layer features to assist in capturing lane line associated information. When detecting the starting point of the lane line, the SOLOv2 method is used to divide the image into Hp*Wp grids, and outputs two feature maps, heatmap and parameter map. The heatmap indicates whether there is a lane line instance in the grid, and the parameter map provides convolution parameters describing the details of the lane line. In the lane line detail description stage, these convolution parameters are applied to the feature maps extracted by FPN to calculate location maps and offset maps. The location maps determine the lane line position through softmax and weighted summation, while the offset maps predict the offset in the horizontal direction to improve the accuracy of the position calculation.

[0018] As a preferred technical solution of the present invention, the estimated vehicle speed adopts a vehicle speed estimation model based on inverse perspective transformation. The inverse perspective transformation is used to eliminate the perspective effect in the image. A 3×3 perspective transformation matrix is ​​solved by at least four non-collinear corresponding point pairs. The edge points of two parallel lane lines are selected as corresponding points, and a mapping relationship between the actual distance and the pixel coordinates is established to estimate the vehicle speed.

[0019] As a preferred technical solution of the present invention, the vehicle detection adopts YOLOv3 as the detection algorithm, and the vehicle tracking adopts the Kalman filter algorithm to update the vehicle's position information in real time, and combine the lane line detection results to predict and analyze the vehicle's driving trajectory.

[0020] Compared with the prior art, the present invention has the following beneficial effects:

[0021] The present invention pre-processes the surveillance video, extracts pictures at alternate frames and adjusts them to a uniform resolution. Lane line annotation is also required in the training stage. After that, FPN is used to perform multi-level feature extraction on the input image, and the features of the high-level semantic level are used to detect the starting point of the lane line, and the low-level semantic features are used to describe the detailed shape of the lane line. After the lane line is detected, the pixel coordinates of the lane line are obtained, and the image is perspectively inverted based on this. The lane line is converted into a bird's-eye view using the perspective inverse transformation, and a mapping relationship is formed in combination with the actual lane line information. Then, the vehicle speed is estimated, providing more reliable driving data for the automatic driving system, which not only improves the accuracy of detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a schematic diagram of the overall process of the present invention;

[0023] Figure 2 Schematic diagram of reverse perspective transformation of lane lines in the present invention. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] like Figure 1 to Figure 2 As shown, the present invention provides a lane line detection system based on FPN under monitoring, comprising:

[0026] Data preparation: pre-process the surveillance video, adjust the image size to 1080P, and also mark the lane lines during the training phase;

[0027] FPN extracts multi-level features and uses FPN to extract multi-level features from the input image;

[0028] The lane line starting point is detected based on the features combined with the conditional convolution kernel, and the features at the high-level semantic level are used to detect the lane line starting point;

[0029] The lane line is described according to the row formula, and the low-level semantic features are used to describe the details of the lane line shape;

[0030] Detect lane lines;

[0031] Perform inverse perspective transformation based on the lane line coordinates. After detecting the lane line, obtain the pixel coordinates of the lane line, and perform inverse perspective transformation on the image based on this.

[0032] Combine pixel coordinates and actual lane line information to build a mapping relationship, and perform vehicle detection and tracking at the same time;

[0033] Estimate vehicle speed.

[0034] Data preparation includes training sets and test sets. Both training sets and test sets are self-made data sets. The video source is video surveillance of highways. The surveillance video is extracted by frame-by-frame extraction to obtain pictures. After extracting the frames, the pictures are first resized to adjust the resolution to 1920×1080, and then the blurred images are manually removed.

[0035] The images in the surveillance video are obtained by extracting alternate frames. Due to the different models and information of the cameras, the images are first resized after the frames are extracted, and the resolution is uniformly adjusted to 1920×1080. Since the surveillance cameras switch between long and short views and the monitoring angle of view during work, there will be blurred images during the changes of these images, so these images need to be manually removed.

[0036] Among them, the lane lines are detected by the lane line detection model based on FPN and conditional convolution kernel, and the lane lines are annotated from left to right using Labelme.

[0037] The lane lines are labeled from left to right using the Labelme tool. Labelme is an open source image annotation tool that allows users to annotate objects in images through simple operations and generate corresponding annotation files. Labelme is used to accurately annotate lane lines to ensure the accuracy of training data.

[0038] Among them, the lane line detection model based on FPN and conditional convolution kernel uses Resnet as the backbone network, and introduces FPN to extract multi-semantic layer features to assist in capturing lane line related information. When detecting the starting point of the lane line, the SOLOv2 method is used to divide the image into Hp*Wp grids, and outputs two feature maps, heatmap and parameter map. The heatmap indicates whether there is a lane line instance in the grid, and the parameter map provides convolution parameters that describe the details of the lane line. In the lane line detail description stage, these convolution parameters are applied to the feature maps extracted by FPN to calculate location maps and offsetmaps. The location maps determine the lane line position through softmax and weighted summation, while the offset maps predict the offset in the horizontal direction to improve the accuracy of the position calculation.

[0039] By using Resnet as the backbone network, since the lane lines are relatively long and thin, a standard FPN is added for multi-semantic layer feature extraction to capture the associated information of the lane lines. In the lane starting point detection part, similar to SOLOv2, the image is divided into several grids (let the number of grids be Hp*Wp), and two feature maps are output, one is a heatmap, which indicates whether there is a lane line instance in each grid, and the other is a parameter map, which corresponds to the heatmap one by one. A set of convolution parameters corresponding to the lane line instance in the grid is output, which is used for the convolution operation in the lane line detail description to calculate the lane line information corresponding to the grid. In the lane line detail description part, first on the feature map extracted by FPN, the convolution parameters calculated previously are used to calculate the location maps and offset maps. The location maps are single-channel feature maps. For each row of data, softmax is first performed, and then the weighted sum of the output of softmax and the column coordinates is performed to obtain the lane line position of the row. Since the position calculation within the grid is not accurate enough, an offset map is used to predict an offset in the horizontal direction.

[0040] Among them, the vehicle speed is estimated using a vehicle speed estimation model based on inverse perspective transformation. The inverse perspective transformation is used to eliminate the perspective effect in the image. A 3×3 perspective transformation matrix is ​​solved through at least four non-collinear corresponding point pairs. The edge points of two parallel lane lines are selected as corresponding points, and the mapping relationship between the actual distance and the pixel coordinates is established to estimate the vehicle speed.

[0041] By using the vehicle speed estimation model based on inverse perspective transformation, in the images taken by the surveillance camera, due to the existence of perspective effect, originally parallel objects intersect in the image, and the inverse perspective transformation is to eliminate this perspective effect. Due to the parameters and position information of the camera, a more convenient way is the transformation method of corresponding points. The input is at least four corresponding point pairs, and there cannot be three or more points collinear. The point pairs are used to solve the perspective transformation matrix, where the transformation matrix is ​​a 3×3 matrix, so a linear equation system can be constructed for solution. The better point pairs are two parallel lane line dashed lines, and the four points on its edge can form a quadrilateral, which meets the input requirements of this transformation method. At the same time, the actual information of this quadrilateral can be better obtained, such as the lane width of the highway is 3.75 meters, and the length of a single lane line dashed line is 6 meters. This information can be used to construct a mapping relationship between the actual distance and the pixel coordinates to estimate the vehicle speed.

[0042] Among them, vehicle detection uses YOLOv3 as the detection algorithm, and vehicle tracking uses the Kalman filter algorithm to update the vehicle's position information in real time, and combine the lane line detection results to predict and analyze the vehicle's driving trajectory.

[0043] By using YOLOv3 as the detection algorithm, vehicles in the video can be identified quickly and accurately. During the vehicle detection and tracking process, YOLOv3 is first used to detect the vehicle in each frame to obtain the vehicle's bounding box and category information. Then, the Kalman filter algorithm is used to track the vehicle to reduce the loss of the target. During the tracking process, the vehicle's position information is updated in real time, and the vehicle's driving trajectory is predicted and analyzed in combination with the lane line detection results.

[0044] The working principle and usage process of the present invention are as follows: the video stream extracts pictures through alternate frames and adjusts them to a uniform resolution. During the extraction process, blurred images are manually removed to ensure processing accuracy. Then, a lane line detection model based on FPN and conditional convolution kernel is used for detection. The model uses the Resnet backbone network and FPN to extract multi-semantic layer features. When detecting the starting point of the lane line, the SOLOv2 method is used to divide the image into multiple grids, and output heatmaps and parameter maps. The heatmap indicates whether there are lane line instances in the grid, and the parameter map provides convolution parameters for lane line details. In the detail description stage, the convolution parameters are applied to calculate location maps and offset maps to determine the lane line position and predict the offset in the horizontal direction. Finally, the vehicle speed is estimated using a model based on inverse perspective transformation, the perspective effect is eliminated through inverse perspective transformation, and a mapping relationship between real distance and pixel coordinates is established to estimate the vehicle speed.

[0045] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0046] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A lane line detection system based on FPN under monitoring, characterized in that: include: Data preparation: pre-process the surveillance video, adjust the image size to 1080P, and also mark the lane lines during the training phase; FPN extracts multi-level features and uses FPN to extract multi-level features from the input image; The lane line starting point is detected based on the features combined with the conditional convolution kernel, and the features at the high-level semantic level are used to detect the lane line starting point; The lane line is described according to the row formula, and the low-level semantic features are used to describe the details of the lane line shape; Detect lane lines; Perform inverse perspective transformation based on the lane line coordinates. After detecting the lane line, obtain the pixel coordinates of the lane line, and perform inverse perspective transformation on the image based on this. Combine pixel coordinates and actual lane line information to build a mapping relationship, and perform vehicle detection and tracking at the same time; Estimate vehicle speed.

2. The lane line detection system based on FPN under monitoring according to claim 1, characterized in that: The data preparation includes a training set and a test set, both of which are self-made data sets. The video source is video surveillance of highways. The surveillance video is extracted by frame-by-frame extraction to obtain pictures. After extracting the frames, the pictures are first resized to uniformly adjust the resolution to 1920×1080, and then the blurred images are manually removed.

3. The lane line detection system based on FPN under monitoring according to claim 1, characterized in that: The detected lane lines are detected by a lane line detection model based on FPN and conditional convolution kernel, and Labelme is used to mark the lane lines from left to right.

4. The lane line detection system based on FPN under monitoring according to claim 1, characterized in that: The lane line detection model based on FPN and conditional convolution kernel adopts Resnet as the backbone network, and introduces FPN to extract multi-semantic layer features to assist in capturing lane line association information. When detecting the starting point of the lane line, the SOLOv2 method is used to divide the image into Hp*Wp grids, and two feature maps, heatmap and parameter map, are output. The heatmap indicates whether there is a lane line instance in the grid, and the parameter map provides convolution parameters that describe the details of the lane line. In the lane line detail description stage, these convolution parameters are applied to the feature maps extracted by FPN to calculate location maps and offset maps. The locationmaps determine the lane line position through softmax and weighted summation, while the offset maps predict the offset in the horizontal direction to improve the accuracy of the position calculation.

5. The lane line detection system based on FPN under monitoring according to claim 1, characterized in that: The vehicle speed estimation adopts a vehicle speed estimation model based on inverse perspective transformation. The inverse perspective transformation is used to eliminate the perspective effect in the image. A 3×3 perspective transformation matrix is ​​solved through at least four non-collinear corresponding point pairs. The edge points of two parallel lane lines are selected as corresponding points, and a mapping relationship between the actual distance and the pixel coordinates is established to estimate the vehicle speed.

6. The lane line detection system based on FPN under monitoring according to claim 1, characterized in that: The vehicle detection adopts YOLOv3 as the detection algorithm, and the vehicle tracking adopts the Kalman filter algorithm to update the vehicle's position information in real time, and predict and analyze the vehicle's driving trajectory in combination with the lane line detection result.

Images