Hybrid anchor lane line detection result fusion method based on attention mechanism

Abstract

The invention provides a hybrid anchor lane line detection result fusion method based on an attention mechanism, and the method comprises the steps: carrying out the feature extraction of an original image, and obtaining a feature map containing high-level semantics; inputting the global feature map into two multi-layer perceptron in parallel for processing, and outputting a data structure shape matched with the row anchor definition and the column anchor definition; post-processing is carried out, and line anchor and column anchor identification results of lane line coordinates are obtained through conversion; taking identification results of the row anchor and the column anchor as input, integrating information of the row anchor and the column anchor through an attention mechanism, distributing different confidence coefficients for the row anchor and the column anchor, and calculating an offset value of an output lane line coordinate; carrying out normalization processing on the confidence coefficient and the deviation value; and fusing lane line detection results of the row anchor and the column anchor to obtain an identification result. The method can adapt to different road environments and weather conditions, has high real-time performance and good recognition precision, and has remarkable advantages in the aspect of improving the accuracy and robustness of lane line detection.

Description

Technical Field

[0001] The present invention relates to the field of intelligent driving technology, and in particular to a hybrid anchor lane line detection result fusion method based on an attention mechanism. Background Art

[0002] With the widespread penetration of artificial intelligence and big data in all walks of life, smart cars have become the strategic direction of the development of the global automotive industry. The autonomous driving function of smart cars is specifically composed of three major systems: environmental perception, decision planning and motion control system. The environmental perception system detects its own state and driving environment through various sensor data. The decision planning system makes behavioral decisions and path planning based on the surrounding environment data obtained by the environmental perception system. The execution control system is responsible for converting the behavior information of the decision planning system into the execution actions of each moving part.

[0003] Among them, the environment perception system of smart cars is responsible for perceiving the surrounding environment and determining its own status information. Lane line detection is an important part of environment perception technology. It plans the feasible domain and safety domain for vehicle operation decision-making, intelligent obstacle avoidance, etc., and is an important basis for smart cars to follow traffic rules. The quality of lane line detection directly affects the later autonomous driving decision-making and motion control, so accurate lane line detection is crucial for the safe operation of smart cars.

[0004] The current mainstream lane line detection technologies include lane line detection methods based on image processing, lane line detection methods based on laser radar, lane line detection methods based on radar and camera fusion, lane line detection methods based on radar and camera fusion, and lane line detection methods based on GPS and maps. The lane line detection method based on image processing mainly uses image processing algorithms to detect and identify lane lines. The lane line detection method based on laser radar scans the ground through laser radar, analyzes and processes point cloud data to obtain lane line position information. The lane line detection method based on radar and camera fusion fuses the data of the two sensors, radar and camera, to obtain more accurate lane line position information and improve the robustness of detection. The lane line detection method based on GPS and maps mainly uses the GPS on the vehicle and map data to detect the position information of the lane line. This method can overcome the image processing technology in certain specific situations (such as poor lighting). This method can detect lane lines more accurately in weak lighting and bad weather. Among the above-mentioned technical means, lane line detection based on LiDAR relies on dense road point cloud data, so high-thread LiDAR is required. However, the current high-thread LiDAR is expensive and difficult to apply in large-scale engineering projects. Therefore, the technology route based on LiDAR cannot be implemented on a large scale in a short period of time and is relatively niche. Lane line detection based on GPS and maps is heavily dependent on high-precision maps. This type of technology is very demanding on the accuracy of GPS and the integrity and timeliness of map data. The freshness of map data requires high maintenance costs, and it is difficult to cover non-hotspot areas and the algorithm has a narrow application domain, resulting in the basic shelving of intelligent perception routes based on high-precision maps.

[0005] In summary, the current mainstream technical route is the image-based lane detection method, which can achieve large-scale and economical engineering applications. According to the different sources of the detected images, it is divided into monocular camera lane detection and binocular camera lane detection. The binocular camera uses the left and right cameras to simultaneously image the object to be measured from different angles, and calculates the position deviation of the corresponding points of the image based on the parallax principle to obtain the distance information of the object to be measured. Combined with the internal and external parameters of the camera, point cloud data with spatial two-dimensional coordinates can be finally obtained. The binocular camera can detect the target distance without identifying the target, and the distance measurement accuracy is much higher than the distance estimated by the monocular camera through deep learning. However, stereo matching in the parallax principle is a complex technical problem, and solving this technical problem requires the use of complex parallel processing algorithms. In order to ensure the robustness of stereo matching in the presence of errors, the algorithm complexity will be further increased sharply. At the same time, the stereo matching algorithm of the binocular camera contains a large number of complex parallel calculations such as multiplication, division, and square root. The general-purpose GPU is not suitable as the main processing chip, lacks specific processor support, and the hardware programming using FPGA is difficult to implement. In addition, in the real world, light sensitivity will cause a large difference in brightness between the left and right camera images, increasing the difficulty and error of stereo matching. Furthermore, the monotonous texture of the object being measured (such as truck compartments with consistent colors and walls with consistent colors) leads to fewer matching features, which increases the difficulty and error of stereo matching. In view of the complex algorithm implementation of the binocular technology route, the high threshold of processing chip technology, and the difficulty in balancing benefits, lane line detection technology based on binocular vision is currently difficult to apply on a large scale.

[0006] Monocular camera lane detection refers to collecting images of the road through a camera. Traditional methods include edge detection, color segmentation, template matching, etc. The edge detection-based method can detect the edge of the image, find the edge pixels, and then fit the lane line through a straight line fitting algorithm; the color segmentation-based method can segment the lane line by selecting a certain color range; the template matching-based method can first build a lane template, and then find the location of the lane line by matching it in the image. The advantages of traditional algorithms are strong logic and interpretability of the processing flow, small data volume and fast speed in the processing flow, but their disadvantages are also obvious. Traditional algorithms require manual adjustment of filter operators and parameters according to the characteristics of the street scenes targeted by the algorithm. The workload of parameter adjustment is large and the robustness is poor. When the driving environment changes significantly, the lane detection effect is poor. At the same time, each method has high limitations. For example, the Hough line detection method is accurate but cannot detect curves. The fitting method can detect curves but is unstable. The affine transformation can detect multiple lanes but has serious interference in cases of occlusion. The perspective transformation operation has some specific requirements for the camera. The image needs to be adjusted before the transformation. In addition, the installation of the camera and the inclination of the road itself will affect the transformation effect. The traditional lane detection method mainly uses the method of fitting image edges and data points. It has a higher recognition accuracy in environments with less interference and smaller environmental differences. In actual applications, due to the variability of vehicle driving environments, traditional detection algorithms are difficult to adapt. Therefore, the deep learning method is the mainstream of lane detection at this stage.

[0007] Lane detection methods based on deep learning can be roughly divided into four categories, namely segmentation-based, anchor-based, key-point-based and parameter prediction-based methods. The segmentation-based detection method mainly achieves the detection purpose by classifying the image pixel by pixel, separating the detection target from the background. Since the segmentation detection method needs to process pixel by pixel, it has the disadvantages of high computational consumption and slow running speed in both training and reasoning. The key-point-based method directly detects the lane instance and divides the instance using post-processing methods. Although the key-point-based method has both flexibility and real-time performance, it has poor ability to build global information and poor recognition accuracy when the lane is occluded. In addition, the key-point-based method is difficult to build global information when dealing with occlusion problems. The parameter-based method uses parameters to model the lane curve and regresses these parameters to detect the lane. This method has fewer parameters that can be regressed, but is more sensitive to the prediction parameters. For example, the error prediction of high-order coefficients may cause the lane shape to change. In addition, since the method using parameter curves is highly dependent on the edge information of the image, the detection accuracy is low in the case of occlusion and light interference.

[0008] Anchor-based methods can be divided into two categories: line anchor-based methods and row anchor-based methods. Line anchor-based methods use predefined line anchors as references to regress precise lanes. Row anchor-based methods strip the entire image, and then determine whether each subdivided cell of a predefined row anchor is a lane line position. Its advantages are low computational complexity, fast detection speed, good detection accuracy in a variety of scenarios, and can better meet real-time and complex scenario requirements. At this stage, there are many anchors with different structures in the research field, and each has its own advantages and disadvantages for lane line detection. Some anchor methods have the disadvantage of repeated feature calculation and processing.

[0009] Anchor is a common target detection method. Its main idea is to generate some image areas for detecting targets in the image, and then detect whether there are targets and predict the location of the targets. Lane recognition is a type of target detection. The popular processing method at this stage is to introduce the idea of ​​anchors and make corresponding changes to adapt to the slender structure of lanes. Therefore, two lane recognition methods, row anchor and column anchor, are generated. The so-called hybrid anchor is the row anchor and column anchor used for lane detection.

[0010] like Figure 1 As shown, in Figure 1 The image is divided into rows of regions by thick lines, and each region is defined as an anchor. Since the row segmentation method is adopted, it is named row anchor. Figure 1 Each arrow on the left side points to a line anchor, and each line anchor is divided into several squares by a dotted line. The recognition result of the line anchor mainly shows three lane line and anchor related information: (1) Determine whether the line anchor contains lane lines; (2) Determine whether the subdivided squares in the line anchor are the intersections of the line anchor and the lane lines; (3) Determine the number of lane lines that belong to the intersection squares of the line anchor and the lane lines. Figure 1 The middle anchor subdivided square is the focus of the lane line and the anchor, where the left square and the right square belong to two lane lines respectively.

[0011] The column anchor structure is similar to the row anchor structure, but the division direction is different. Figure 2 As shown in the figure, the image is divided into columns of regions by thick lines. Each region is defined as an anchor. Since the column segmentation method is adopted, it is named column anchor. Figure 2 Each arrow above points to a column anchor. In each column anchor, dotted lines are used to divide the anchor into several squares. The information contained in the recognition result of the column anchor is the same as that of the row anchor.

[0012] Taking the lane line detection task of three lanes as an example, the smart car needs to detect four lane lines (1, 2, 3, 4), such as Figure 3The four standard lane lines connected by the middle circle. Since the two lane lines (2 and 3) on the left and right sides of the lane where the vehicle is located are relatively vertical, in the detection method using anchors, the number of intersections between lane lines 2 and 3 and the row anchors is large, and the number of intersections with the column anchors is small. In theory, the larger the amount of information input based on the equal algorithm, the more accurate the detection effect can be obtained. Therefore, in the row anchor mode, lane lines 2 and 3 can obtain more comprehensive information elements, thereby obtaining more accurate detection results than in the column anchor mode. Similarly, the left lane line 1 of the left lane of the vehicle's lane and the right lane line 4 of the right lane of the vehicle's lane are relatively horizontal. In the detection method using anchors, the number of intersections between lane lines 1 and 4 and the row anchors is small, and the number of intersections with the column anchors is large. Therefore, the detection accuracy of lane lines 1 and 4 using column anchors is higher, and the detection accuracy using row anchors is lower.

[0013] The existing technology uses row anchors to detect lane lines in the middle of the image and uses column anchors to detect lane lines on both sides of the image. However, in actual situations, the vehicle's own posture and the camera installation angle will affect the angle of the lane lines in the image. Therefore, the mixed use of row anchors and column anchors may not necessarily detect relatively vertical lane lines.

[0014] exist Figure 3 In the figure, the lane lines composed of small dots represent the lane line detection results using anchors. In the lane line detection process of row anchors and column anchors, the prediction results between row anchors (or between column anchors) are not referenced and associated with each other, and the information dimension is not fully utilized. Therefore, in the red area, there are a large number of irregular green lane line prediction points that deviate greatly from the true value. Summary of the invention

[0015] In view of the shortcomings of existing anchor point-based lane line detection, the purpose of the present invention is to provide a hybrid anchor lane line detection result fusion method based on attention mechanism, which can adapt to different road environments and weather conditions, such as illumination changes, occlusions, shadows, etc., has high real-time performance and good recognition accuracy, and has significant advantages in improving the accuracy and robustness of lane line detection.

[0016] The present invention achieves the above-mentioned purpose through the following technical solutions:

[0017] A hybrid anchor lane detection result fusion method based on an attention mechanism, the method comprising the following steps:

[0018] The backbone model is used to extract features from the original image to obtain a feature map containing high-level semantics;

[0019] The global feature map is input into two multi-layer perceptrons in parallel for processing. Each multi-layer perceptron processes the feature map separately and outputs a data structure shape that matches the row anchor and column anchor definitions.

[0020] Post-process the row anchor and column anchor data structures containing lane line information output by the multi-layer perceptron and transform them into row anchor and column anchor recognition results of lane line coordinates;

[0021] Define a fully connected layer based on the attention mechanism, taking the recognition results of row anchors and column anchors as input;

[0022] The attention mechanism is used to integrate the information of row anchors and column anchors, assign different confidence levels to them, and calculate the offset value of the output lane line coordinates.

[0023] Normalize the confidence and offset values ​​to constrain their data ranges;

[0024] The normalized confidence and offset values ​​are used to fuse the lane line detection results of the row anchor and column anchor to obtain the final lane line recognition result.

[0025] According to a hybrid anchor lane detection result fusion method based on an attention mechanism provided by the present invention, the feature extraction of the original image through the backbone model includes:

[0026] Preprocess the original image to meet the input requirements of the model;

[0027] Select a backbone model for image feature extraction to gradually extract features from the image;

[0028] The last convolutional layer of the backbone model outputs a feature map, which retains the spatial structure of the original image, but the value at each position represents the feature strength at that position, that is, a global feature map containing high-level semantics;

[0029] The global feature map is input into two multi-layer perceptrons in parallel for processing, and the output data structure shape matches the row anchor and column anchor definitions;

[0030] Through post-processing, the output of the multi-layer perceptron is converted into a sequence of lane line coordinate points of indefinite length detected by row anchors and column anchors.

[0031] According to a hybrid anchor lane line detection result fusion method based on an attention mechanism provided by the present invention, a fixed number of lane line coordinate points are generated by interpolation or fixed step selection of a lane line coordinate point sequence of indefinite length;

[0032] Assuming that the number of lane line coordinate points generated is l, the resulting coordinates of the row anchor and column anchor are as follows:

[0033]

[0034] Where r represents the lane line coordinates identified by the anchor, c represents the lane line coordinates identified by the anchor, i is the sequence number of the coordinate point, and the value range is 1≤i≤l, x and y represent the horizontal and vertical coordinates respectively;

[0035] The row anchor and column anchor recognition results of the output lane line coordinates are P r and P c .

[0036] According to a hybrid anchor lane detection result fusion method based on an attention mechanism provided by the present invention, a fully connected layer with learnable parameters is defined. The fully connected layer takes the recognition results of row anchors and column anchors as input and takes a weight matrix and a coordinate offset matrix as output.

[0037] According to a hybrid anchor lane line detection result fusion method based on an attention mechanism provided by the present invention, W is defined r Is the row anchor output result P r The weight, W c Is the row anchor output result P c The weight of , Δ is the coordinate offset matrix, that is, W r , W c and Δ are the outputs of the attention mechanism, W r , W c The resulting expression of and Δ is the following formula:

[0038]

[0039] According to a hybrid anchor lane detection result fusion method based on an attention mechanism provided by the present invention, a lane anchor recognition point is defined and column anchor identification points Since the output of the fully connected layer as well as Its value range is between (-∞, +∞), so it needs to be normalized to ensure that the calculation results will not deviate far from the normal value;

[0040] For the weight parameter For example, the two represent points and Point The proportion coefficient in the result is determined by soft normalization (SoftMax). The calculation formula of the proportion coefficient of the row anchor recognition result is expressed as follows:

[0041]

[0042] Among them, s represents the normalized result, and the calculation formula of the proportion coefficient of the column anchor recognition result is expressed as follows:

[0043]

[0044] So the weight matrix W r and W c The normalized result W rs and W cs The following formula:

[0045]

[0046] According to a hybrid anchor lane detection result fusion method based on an attention mechanism provided by the present invention, for the offset matrix, since it is a single value, the Sigmod function is used for normalization and scaling to (-1, +1), and the calculation formula for normalizing and scaling the offset value at the coordinate point labeled i is expressed as the following formula:

[0047]

[0048] Therefore, the normalized result of the offset matrix Δ is expressed as the following formula:

[0049]

[0050] According to a hybrid anchor lane detection result fusion method based on an attention mechanism provided by the present invention, when expanding the offset, the offset magnification factor adopts the coordinate difference between the row anchor and column anchor recognition results. Considering the offset matrix Δ, the row anchor and column anchor fusion results (x i ,y i ) The calculation formula is expressed as the following formula:

[0051]

[0052] The final coordinate matrix P of the fusion row anchor and column anchor recognition results is obtained, which is expressed as the following formula:

[0053] P=[(x 1 ,y 1 ),(x 2 ,y 2 ),…,(x i ,y i ),…,(x l ,y l )].

[0054] According to a hybrid anchor lane line detection result fusion method based on an attention mechanism provided by the present invention, the method of obtaining the final lane line recognition result includes:

[0055] The final lane line recognition result consists of the recognition results of row anchors and column anchors, weight parameters, and coordinate offset matrix, which can be expressed as the following formula:

[0056] P=W rs *P r +W cs *P c +Δ s *|P r -P c |.

[0057] According to a hybrid anchor lane detection result fusion method based on an attention mechanism provided by the present invention, the following steps are further performed:

[0058] Estimate the uncertainty of the model's lane detection results and adjust the confidence or perform post-processing based on the uncertainty;

[0059] According to the estimated uncertainty, one or more thresholds are set. When the uncertainty exceeds a certain threshold, the confidence of the corresponding lane line detection result is reduced.

[0060] For lane line detection results with too low confidence, choose to remove or smooth them to reduce false detections or improve the smoothness of the detection results.

[0061] It can be seen that compared with the prior art, the method provided by the present invention has the following beneficial effects:

[0062] In the present invention, lane line hybrid anchor detection uses two parallel multi-layer perceptrons to implement the classification tasks of row anchors and column anchors on the global feature map, so that the model allows row anchors and column anchors to perform lane line detection on the entire image, avoiding the row anchor or column anchor only detecting the local part of the image and missing the lane line that is relatively perpendicular to the anchor, giving full play to the lane line detection advantages of row anchors and column anchors. At the same time, the two multi-layer perceptrons corresponding to the row anchor and column anchor respectively use the same feature map as input to reduce the amount of feature extraction calculations, and parallel calculations and recognition results without interfering with each other, so that the speed and real-time performance of lane line recognition can be met in real-life scenarios.

[0063] In the present invention, lane line hybrid anchor detection uses two parallel multi-layer perceptrons to perform lane line detection on the global feature map using the row anchor and column anchor mechanisms respectively. Different from the traditional row anchor that only detects the middle of the image and the column anchor that only detects the two sides of the image, the use of a parallel global detection method not only ensures the detection speed, but also ensures that the row anchor or column anchor can detect lane lines that are more vertical relative to itself.

[0064] In the present invention, the result fusion under the attention mechanism takes the lane line recognition results of the row anchor and the column anchor as input, distributes the confidence by comprehensively analyzing the information of the two, and calculates the offset value of the output lane line coordinate, and finally obtains the final result of lane line recognition. The lane line detection results of the row anchor and the column anchor are fused, which not only avoids the situation where large errors occur when only row anchors or column anchors are used, but also makes the recognition results between adjacent anchors in the row anchor or column anchor refer to each other, avoids the occurrence of irregular lane line prediction points, and improves the accuracy and stability of lane line detection.

[0065] In the present invention, the attention mechanism is used to fuse the lane line recognition results of the row anchor and the column anchor. Not only the confidence is assigned to the lane line recognition results of the row anchor and the column anchor, but also the coordinate offset value of the lane line point is calculated for further correction. The recognition advantages of the row anchor and the column anchor are fused, thereby improving the stability and reliability of the lane line recognition system.

[0066] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 It is a schematic diagram of the anchor used in lane line detection in the prior art.

[0068] Figure 2 This is a schematic diagram of the column anchors used in lane line detection in the prior art.

[0069] Figure 3 It is a schematic diagram comparing lane lines detected using row anchors and column anchors with four standard lane lines in the prior art.

[0070] Figure 4 It is a flowchart of an embodiment of a hybrid anchor lane line detection result fusion method based on an attention mechanism of the present invention.

[0071] Figure 5 It is a schematic diagram of hybrid anchor parallel detection of lane lines in an embodiment of a hybrid anchor lane line detection result fusion method based on an attention mechanism of the present invention.

[0072] Figure 6 It is a schematic diagram of the result fusion of the attention mechanism in an embodiment of a hybrid anchor lane line detection result fusion method based on the attention mechanism of the present invention. DETAILED DESCRIPTION

[0073] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0074] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0075] See also Figures 4 to 6 The present invention provides a hybrid anchor lane line detection result fusion method based on an attention mechanism, the method comprising the following steps:

[0076] Step S1, extracting features from the original image through the backbone model to obtain a feature map containing high-level semantics;

[0077] Step S2, inputting the global feature map into two multi-layer perceptrons in parallel for processing, each multi-layer perceptron processes the feature map separately and outputs a data structure shape that matches the row anchor and column anchor definitions;

[0078] Step S3, post-processing the row anchor and column anchor data structures containing lane line information output by the multi-layer perceptron to convert them into row anchor and column anchor recognition results of lane line coordinates;

[0079] Step S4, defining a fully connected layer based on the attention mechanism, taking the recognition results of the row anchors and column anchors as input;

[0080] Step S5: Integrate the information of row anchors and column anchors through the attention mechanism, assign different confidences to them, and calculate the offset value of the output lane line coordinates;

[0081] Step S6, normalizing the confidence and offset values ​​to constrain their data ranges;

[0082] Step S7, using the normalized confidence and offset values ​​to fuse the lane line detection results of the row anchor and the column anchor, and obtain and apply the final lane line recognition result.

[0083] The hybrid anchor lane detection result fusion method based on the attention mechanism provided in this embodiment has several significant advantages, mainly including:

[0084] Efficient feature extraction: By extracting features from the original image through the backbone model, a feature map with high-level semantics can be obtained, which provides rich and accurate information for subsequent anchor point definition and lane line detection.

[0085] Parallel multi-layer perceptron processing: Processing the global feature map in parallel can significantly improve computational efficiency. At the same time, two multi-layer perceptrons perform customized processing for row anchors and column anchors respectively, so that the output data structure shape matches the anchor definition, improving recognition accuracy.

[0086] Introduction of attention mechanism: The attention mechanism can integrate the information of row anchors and column anchors and assign different confidence levels to them. This mechanism enables the model to focus on more important information and ignore irrelevant or noisy information, thereby improving the accuracy of lane line detection.

[0087] Calculation of coordinate offset value: Based on the original coordinate information, the attention mechanism is used to calculate the offset value of the output lane line coordinate. This method can adjust the position of the lane line more finely and improve the accuracy of lane line recognition.

[0088] Fusion and optimization of results: The lane detection results of row anchors and column anchors are fused using the normalized confidence and offset values ​​to obtain the final lane recognition result. This fusion method can integrate the information of the two anchor points and further improve the robustness and accuracy of recognition.

[0089] Wide application: This method is not only suitable for lane line detection, but can also be extended to other similar scenarios, such as safety warnings, assisted driving, traffic monitoring, etc., and has wide applicability.

[0090] Strong interpretability: Since this method is based on anchor points and attention mechanisms, its output results are interpretable to a certain extent, which facilitates subsequent analysis and optimization.

[0091] Strong adaptability: This method can adapt to different road environments and weather conditions, such as lighting changes, occlusions, shadows, etc., and has strong robustness.

[0092] In the above step S1, feature extraction is performed on the original image through the backbone model, including:

[0093] Preprocess the original image, such as resizing and normalizing, to meet the input requirements of the model;

[0094] Select a backbone model for image feature extraction, such as ResNet, VGG, MobileNet, etc. It contains multiple convolutional layers, pooling layers, activation functions, etc., which are used to gradually extract features from the image;

[0095] As the image passes through different layers in the backbone model, the network learns features from low-level to high-level. Low-level features are usually basic information such as edges, colors, textures, etc., while high-level features are more abstract and have semantic information.

[0096] The last convolutional layer of the backbone model outputs a feature map, which retains the spatial structure of the original image, but the value at each position represents the feature intensity at that position, that is, a global feature map containing high-level semantics.

[0097] It can be seen that in this embodiment, the original image is first feature extracted through a backbone model (such as a residual network), irrelevant information of the image is removed, noise information is reduced, and a feature map containing high-level semantics is obtained. Since the original image contains a large amount of information, and more irrelevant information, the signal-to-noise ratio is small and the data processing volume is large; at the same time, the information within each anchor does not refer to each other, and there is no global information as a reference. Therefore, the anchor of this embodiment is not defined on the original image, but on the feature map.

[0098] In the above step S2, the feature map is the extraction of the main features of the original image. First, the row anchor and column anchor are used to detect the image globally, and the lane line detection results of the row anchor and column anchor are output in parallel, that is, the global feature map is input into two multi-layer perceptrons in parallel for processing, and the data structure shape matching the row anchor and column anchor definitions is output. In step S3, the output of the multi-layer perceptron is converted into a sequence of lane line coordinate points of indefinite length detected by the row anchor and column anchor through post-processing, such as Figure 5 shown.

[0099] In the above step S3, in post-processing, since the number of lane line coordinate points generated by row anchors and column anchors is uncertain, and the data input scale of the subsequent attention mechanism is certain, the indefinite length lane line coordinate point sequence is interpolated or selected with a fixed step to generate a fixed number of lane line coordinate points.

[0100] Assuming that the number of lane line coordinate points generated is l, the resulting coordinates of the row anchor and column anchor are as follows:

[0101]

[0102] Where r represents the lane line coordinates identified by the anchor, c represents the lane line coordinates identified by the anchor, i is the sequence number of the coordinate point, and the value range is 1≤i≤l, x and y represent the horizontal and vertical coordinates respectively;

[0103] The row anchor and column anchor recognition results of the output lane line coordinates are P r and P c .

[0104] like Figure 6As shown, this embodiment defines a fully connected layer with learnable parameters based on the attention mechanism. The fully connected layer takes the recognition results of row anchors and column anchors as input and outputs a weight matrix and a coordinate offset matrix.

[0105] Define W r Is the row anchor output result P r The weight, W c Is the row anchor output result P c The weight of , Δ is the coordinate offset matrix, that is, W r , W c Both W and Δ are the outputs of the attention mechanism. The weight parameter is calculated from the global information and is the confidence of the output result. The information of the two is integrated by weighting the row anchor and column anchor output results through the weight parameter. The coordinate offset matrix is ​​added on the basis of the former to make corrections to avoid the recognition results being distributed only between the output results of the row anchor and the column anchor. r , W c The resulting expression of and Δ is the following formula:

[0106]

[0107] In this embodiment, the row anchor identification point and column anchor identification points As an example, the process of recognition result fusion is described in detail. as well as Its value range is between (-∞, +∞), so it needs to be normalized to ensure that the calculation results do not deviate far from the normal value.

[0108] For the weight parameter For example, the two represent points and Point The proportion coefficient in the result is determined by soft normalization (SoftMax). The calculation formula of the proportion coefficient of the row anchor recognition result is expressed as follows:

[0109]

[0110] Among them, s represents the normalized result, and the calculation formula of the proportion coefficient of the column anchor recognition result is expressed as follows:

[0111]

[0112] So the weight matrix W r and W c The normalized result W rs and W cs The following formula:

[0113]

[0114] In this embodiment, since the offset matrix is ​​a single value, the Sigmod function is used for normalization and scaling to (-1, +1). The calculation formula for normalizing and scaling the offset value at the coordinate point labeled i is expressed as follows:

[0115]

[0116] Therefore, the normalized result of the offset matrix Δ is expressed as the following formula:

[0117]

[0118] In order to further expand the offset, the offset magnification factor uses the coordinate difference between the row anchor and column anchor recognition results. Considering the offset matrix Δ, the row anchor and column anchor fusion results (x i ,y i ) The calculation formula is expressed as the following formula:

[0119]

[0120] Therefore, the coordinate matrix P of the final fusion row anchor and column anchor recognition results can be obtained, which is expressed as the following formula:

[0121] P=[(x 1 ,y 1 ),(x 2 ,y 2 ),…,(x i ,y i ),…,(x l ,y l )]

[0122] The above process is the row anchor identification point and column anchor identification points Calculate the coordinate point of the fusion result (x i ,y i ) can be extended to the calculation process of other points. The calculation process of each result point is parallel, so the speed and real-time performance of the algorithm can be guaranteed. In order to facilitate the description of the data calculation process, Figure 5 The normalization process of weight parameters and offset matrix is ​​not shown in the figure. The final output result consists of the recognition results of row anchors and column anchors, weight parameters and coordinate offset matrix. The calculation formula is as follows:

[0123] P=W rs *P r +W cs *P c +Δ s *|P r -P c |

[0124] In this embodiment, the above Figure 5 In the calculation from the feature map to the row anchor data structure, in addition to using the multi-layer perceptron, large-scale convolution kernel convolution can also be used, as well as the serial processing method of large-scale convolution kernel convolution and multi-layer perceptron can be used to replace the multi-layer perceptron to achieve the classification task.

[0125] In this embodiment, the above Figure 6 In the figure, the attention mechanism that integrates the row anchor and column anchor recognition results is implemented using a fully connected layer. In addition to the fully connected layer, recurrent neural networks, self-attention mechanisms, multi-head attention mechanisms, spatial attention mechanisms, etc. can also be used instead of the fully connected layer to achieve weight output.

[0126] In this embodiment, the following is also performed:

[0127] Estimate the uncertainty of the model's lane detection results and adjust the confidence or perform post-processing based on the uncertainty;

[0128] According to the estimated uncertainty, one or more thresholds are set. When the uncertainty exceeds a certain threshold, the confidence of the corresponding lane line detection result is reduced.

[0129] For lane line detection results with too low confidence, choose to remove or smooth them to reduce false detections or improve the smoothness of the detection results.

[0130] Specifically, uncertainty can be effectively estimated by using learning-based methods (such as 3D-LaneNet) or Monte Carlo methods.

[0131] Using the 3D-LaneNet method: This method is a 3D lane detection method based on a semi-local BEV (bird's eye view) grid representation, which combines a parameterized model for segment learning and clustering segments into deep feature embeddings for full lanes. This method can output a Gaussian distribution of each lane segment parameter and estimate its mean and variance values ​​to achieve uncertainty estimation. The network consists of two processing pipelines: image view and bird's eye view (BEV) view. The image view encoder consists of ResNet blocks, while the BEV backbone consists of a feature map of the projected image view and is concatenated with the convolutional projection feature map from the previous block. The input of the network is a monocular image, and the feature map is projected to a bird's eye view (BEV) through a dual-pathway backbone method.

[0132] Monte Carlo method: The uncertainty of the model is estimated by randomly sampling the input to obtain the distribution of the model's prediction results. Multiple predictions are made for the same input, and the variability of the prediction results is observed to estimate the uncertainty.

[0133] In this embodiment, the hybrid anchor lane line detection result fusion method involved in the present invention gives full play to the advantages of lane line detection of row anchors and column anchors, and further improves the recognition accuracy of lane lines and the stability of the detection system. First, the lane line detection method of row anchors and column anchors is used to detect the entire image, and the detection results of the image are output respectively. Performing lane line detection of row anchors and column anchors on the entire image can avoid the row anchor only recognizing the middle of the image and missing the more vertical lane lines on both sides of the image, and can also avoid the column anchor only recognizing the two sides of the image and missing the more horizontal lane lines in the middle of the image.

[0134] In this embodiment, after obtaining the lane line recognition results of the row anchor and the column anchor, the present invention uses the attention mechanism to fuse the two recognition results and output the final fused recognition result. The core idea of ​​the attention mechanism is to select a confidence for each information by calculating the similarity of each information. The so-called confidence is also a weight, which indicates the proportion of the current result in the final output result. The attention mechanism fuses the global information to assign weights to the lane line coordinates output by each anchor, and outputs the offset value of the corresponding coordinate value. The lane line output results of the row anchor and the column anchor and the corresponding weights and coordinate offset values ​​are used to output the final fusion result. The fusion method using the attention mechanism can fuse the lane line recognition results of the row anchor and the column anchor, fully combining the advantages of the row anchor and the column anchor in lane line recognition, and further improving the recognition accuracy. At the same time, the recognition results between adjacent anchor frames under the row anchor or column anchor recognition mechanism can be linked to make the coordinate distribution of the output lane line close to a straight line, rather than an irregular random distribution.

[0135] In practical applications, lane detection is the process of identifying and tracking lane lines on the road using computer vision technology. This technology has many potential use cases, including but not limited to the following:

[0136] 1. Autonomous driving technology: Lane detection is a vital part of autonomous vehicles. By monitoring lane lines in real time, vehicles can stay in lanes more accurately, avoiding collisions and accidents with other vehicles; 2. Safety warning system: Lane detection can also be used to implement the vehicle's departure warning system. When the vehicle accidentally deviates from the lane, the system can issue an alarm to remind the driver, reducing the risk of traffic accidents caused by driver fatigue, distraction or other reasons; 3. Assisted driving function: Many modern vehicles are equipped with lane keeping assist systems, which use lane detection technology to help drivers keep the vehicle in the appropriate lane; 4. Traffic monitoring: In urban traffic management, lane detection can be used to monitor traffic flow, identify congestion and violations, and help traffic management departments make more effective transportation planning and management decisions; 5. Automatic parking: Many automatic parking systems use lane detection technology to identify parking spaces and locate vehicles, thereby realizing the function of automatic parking; 6. Road marking maintenance: Lane detection technology can be used to monitor the wear and damage of road markings to help traffic management departments maintain and update road markings in a timely manner.

[0137] In general, the lane detection technology used in the present invention has broad application prospects in improving traffic safety, optimizing traffic management and developing intelligent transportation systems.

[0138] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0139] The above-mentioned embodiments are only preferred embodiments of the present invention and cannot be used to limit the scope of protection of the present invention. Any non-substantial changes and substitutions made by technicians in this field on the basis of the present invention shall fall within the scope of protection required by the present invention.

Claims

1. A hybrid anchor lane detection result fusion method based on attention mechanism, characterized in that: The method comprises the following steps: The backbone model is used to extract features from the original image to obtain a feature map containing high-level semantics; The global feature map is input into two multi-layer perceptrons in parallel for processing. Each multi-layer perceptron processes the feature map separately and outputs a data structure shape that matches the row anchor and column anchor definitions. Post-process the row anchor and column anchor data structures containing lane line information output by the multi-layer perceptron and transform them into row anchor and column anchor recognition results of lane line coordinates; Define a fully connected layer based on the attention mechanism, taking the recognition results of row anchors and column anchors as input; The attention mechanism is used to integrate the information of row anchors and column anchors, assign different confidence levels to them, and calculate the offset value of the output lane line coordinates. Normalize the confidence and offset values ​​to constrain their data ranges; The normalized confidence and offset values ​​are used to fuse the lane line detection results of the row anchor and column anchor to obtain the final lane line recognition result.

2. The method according to claim 1, characterized in that The feature extraction of the original image by using the backbone model includes: Preprocess the original image to meet the input requirements of the model; Select a backbone model for image feature extraction to gradually extract features from the image; The last convolutional layer of the backbone model outputs a feature map, which retains the spatial structure of the original image, but the value at each position represents the feature strength at that position, that is, a global feature map containing high-level semantics; The global feature map is input into two multi-layer perceptrons in parallel for processing, and the output data structure shape matches the row anchor and column anchor definitions; Through post-processing, the output of the multi-layer perceptron is converted into a sequence of lane line coordinate points of indefinite length detected by row anchors and column anchors.

3. The method according to claim 1, characterized in that: The indefinite length lane line coordinate point sequence is interpolated or selected with a fixed step size to generate a fixed number of lane line coordinate points; Assuming that the number of lane line coordinate points generated is l, the resulting coordinates of the row anchor and column anchor are as follows: Where r represents the lane line coordinates identified by the anchor, c represents the lane line coordinates identified by the anchor, i is the sequence number of the coordinate point, and the value range is 1≤i≤l, x and y represent the horizontal and vertical coordinates respectively; The row anchor and column anchor recognition results of the output lane line coordinates are P r and P c .

4. The method according to claim 3, characterized in that: Based on the attention mechanism, a fully connected layer with learnable parameters is defined. The fully connected layer takes the recognition results of row anchors and column anchors as input and outputs the weight matrix and coordinate offset matrix.

5. The method according to claim 4, characterized in that: Define W r Is the row anchor output result P r The weight, W c Is the row anchor output result P c The weight of , Δ is the coordinate offset matrix, that is, Q r , W c and Δ are the outputs of the attention mechanism, W r , W c The resulting expression of and Δ is the following formula:

6. The method according to claim 5, characterized in that: Define row anchor identification points and column anchor identification points Since the output of the fully connected layer as well as Its value range is between (-∞, +∞), so it needs to be normalized to ensure that the calculation results will not deviate far from the normal value; For the weight parameter and For example, the two represent points and Point The proportion coefficient in the result is determined by soft normalization (SoftMax). The calculation formula of the proportion coefficient of the row anchor recognition result is expressed as follows: Among them, s represents the normalized result, and the calculation formula of the proportion coefficient of the column anchor recognition result is expressed as follows: So the weight matrix W r and W c The normalized result W rs and W cs The following formula:

7. The method according to claim 6, characterized in that: For the offset matrix, since it is a single value, the Sigmod function is used for normalization and scaling to (-1, +1). The calculation formula for normalization and scaling of the offset value at the coordinate point labeled i is expressed as follows: Therefore, the normalized result of the offset matrix Δ is expressed as the following formula:

8. The method according to claim 7, characterized in that: When expanding the offset, the offset magnification factor uses the coordinate difference between the row anchor and column anchor recognition results. and Considering the offset matrix Δ, the row anchor and column anchor fusion results (x i ,y i ) The calculation formula is expressed as the following formula: The final coordinate matrix P of the fusion row anchor and column anchor recognition results is obtained, which is expressed as the following formula: P=[(x1,y1),(x2,y2),…,(x i ,y i ),…,(x l ,y l )]。 9. The method according to claim 8, characterized in that The final lane line recognition result is obtained, including: The final lane line recognition result consists of the recognition results of row anchors and column anchors, weight parameters, and coordinate offset matrix, which can be expressed as the following formula: P=W rs *P r +W cs *P c +Δ s *|P r -P c |。 10. The method according to claim 1, characterized in that Also execute: Estimate the uncertainty of the model's lane detection results and adjust the confidence or perform post-processing based on the uncertainty; According to the estimated uncertainty, one or more thresholds are set. When the uncertainty exceeds a certain threshold, the confidence of the corresponding lane line detection result is reduced. For lane line detection results with too low confidence, choose to remove or smooth them to reduce false detections or improve the smoothness of the detection results.

Images