A method and system for generating three-dimensional lane lines by integrating contextual semantic information

Abstract

The present invention belongs to the field of image processing technology, and specifically relates to a method and system for generating three-dimensional lane lines by integrating contextual semantic information, which obtains relatively dense lane line data of multiple frames of images. Based on a projection-sampling strategy, a first coordinate transformation is first performed to transform the anchor point to be transformed from the ground coordinate system of the current frame to the ground coordinate system of the historical frame, and then a second coordinate transformation is performed to transform the anchor point in the ground coordinate system to the front view. Combined with the information of the historical frame, the coordinate transformations of different frames can more accurately match the lane lines. Then, the temporal context fusion is completed through the cross-attention mechanism processing. The fused feature map can integrate the information of different time frames, provide rich context information, better understand the lane line structure and morphology of the current frame, and enhance the accuracy of the three-dimensional lane line generation.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a method and system for generating three-dimensional lane lines by integrating contextual semantic information. Background Art

[0002] In recent years, deep learning technology has maintained rapid development, and its application in artificial intelligence has developed even more rapidly. Among them, the automatic driving technology that is standard for new energy vehicles is undoubtedly the culmination of artificial intelligence applications, and the accurate recognition of lane lines is related to the success or failure of automatic driving technology. At the same time, current research on lane line detection does not use the method of similar features of adjacent images in lane line data sets.

[0003] Traditional 2D lane detection methods mainly include edge detection and Hough transform, color transform and perspective transform. Edge detection plus Hough transform identifies lane lines through grayscale processing and Canny operator, but has poor real-time performance and insufficient robustness under complex road conditions. Color transform depends on the setting of color space thresholds, is sensitive to changes in lighting, and is easily affected by the colors of surrounding vehicles. Perspective transform converts the front view to the bird's-eye view (BEV) space for prediction. Although it improves efficiency, it relies too much on ideal road information and often ignores contextual background, resulting in poor performance on complex roads, which may affect driving safety. Overall, existing algorithms face many challenges in dealing with actual road conditions. Summary of the invention

[0004] The present invention provides a method and system for generating three-dimensional lane lines by integrating contextual semantic information.

[0005] The technical solution of the present invention is as follows:

[0006] The present invention provides a method for generating three-dimensional lane lines by integrating contextual semantic information, comprising the following steps:

[0007] S1: Obtain image data of a preset number of frames and mark lane lines;

[0008] The image data includes a current frame and a historical frame;

[0009] S2: With the camera as the reference, the intersection of the vertical line extending toward the ground and perpendicular to the ground plane and the ground plane is taken as the coordinate origin to construct the ground coordinate system;

[0010] Based on the ground coordinate system, the three-dimensional coordinate points of the ground of the current frame image and the historical frame image are obtained, and after three-dimensional anchor sampling, the three-dimensional feature anchor point position of the current frame and the three-dimensional feature anchor point position of the historical frame are obtained;

[0011] Based on the camera external parameters of the current frame image and the historical frame images, the position change of each anchor point in the current frame is calculated;

[0012] Based on the position change, the three-dimensional feature anchor point position of the current frame is transformed into the ground coordinate system of the historical frame through the first coordinate transformation to obtain the transformed anchor point position;

[0013] The camera coordinate system is constructed with the camera as the coordinate origin. Based on the camera intrinsic parameters, the transformed anchor point position is projected to the camera coordinate system after the second coordinate transformation to obtain the two-dimensional feature point position of the current frame;

[0014] Based on the two-dimensional feature point positions of the current frame, the front view features of the current frame are obtained;

[0015] The 3D feature anchor point position of the historical frame is projected to the camera coordinate system after the second coordinate transformation to obtain the front view feature of the historical frame;

[0016] S3: The front view features of the current frame and the front view features of the historical frames are processed by the cross attention mechanism to obtain a fused feature map;

[0017] S4: Fusion feature maps are used for classification and regression processing to generate three-dimensional lane lines.

[0018] In step S2, based on the position change, the three-dimensional feature anchor point position of the current frame is transformed by a first coordinate transformation to obtain a transformed anchor point position, which is specifically:

[0019] According to the formula: , get the transformed anchor point position;

[0020] In the formula, x ´、 y ´、 z ' is the anchor point after transformation x axis, y axis, z The coordinate values ​​of the axis; x , y , z is the 3D feature anchor point of the current frame x axis, y axis, z The coordinate values ​​of the axis; Tx , Ty , Tz Respectively represent x , y , z exist x direction, y direction, z The change in position of the direction; is the feature transformation matrix.

[0021] In step S2, a camera coordinate system is constructed with the camera as the coordinate origin. Based on the camera intrinsic parameters, the transformed anchor point position is projected to the camera coordinate system after a second coordinate transformation to obtain the two-dimensional feature point position of the current frame, specifically:

[0022] According to the formula: , get the two-dimensional feature point position of the current frame;

[0023] In the formula, u k , v k is the two-dimensional feature point of the current frame u , v The coordinate values ​​of the axis; Anchor k The vertical distance to the plane where the camera is located, and the plane where the camera is located is perpendicular to the direction of camera movement; K is the camera internal parameter; Represents the transformation matrix from the ground coordinate system to the camera coordinate system; , , Anchor k exist x axis, y axis, z The coordinate value after axis transformation.

[0024] In step S2, based on the camera extrinsic parameters of the current frame image and the historical frame image, the position change of each anchor point in the current frame is calculated, specifically:

[0025] According to the formula: , get the position change of each anchor point in the current frame;

[0026] In the formula, ( Tx , Ty , Tz ) represents the position change of each anchor point in the current frame; R2 is the rotation matrix of the historical frame; t1 and t2 are the displacement vectors of the current frame and the historical frame respectively.

[0027] In step S3, the front view features of the current frame and the front view features of the historical frames are processed by a cross attention mechanism to obtain a fused feature map, and the operation is as follows:

[0028] The front view features of the current frame are converted into a query matrix, and the front view features of the historical frames are converted into a key matrix and a value matrix. The dot product of the query matrix and the key matrix is ​​used to calculate the attention weight. After the dot product operation of the attention weight and the value matrix, the sum is obtained to obtain a fused feature map.

[0029] Furthermore, the front view features of the current frame and the front view features of the historical frames are processed by the cross attention mechanism to obtain a fused feature map, specifically:

[0030] For straight lane lines, the preset number of frames is two. The front view features of the current frame are converted into a query matrix, and the front view features of a certain frame in the history frame are converted into a key matrix and a value matrix. After being processed by the cross attention mechanism, the obtained fusion features are recorded as a fusion feature map.

[0031] For curved lane lines, if the preset number of frames is more than two frames, the front view features of the current frame are converted into a query matrix, and the front view features of the first frame before the current frame are converted into a key matrix and a value matrix, which are processed by a cross-attention mechanism to obtain a first fused feature; the first fused feature is converted into a query matrix, and the front view features of the second frame before the current frame are converted into a key matrix and a value matrix, which are processed by a cross-attention mechanism to obtain a second fused feature; the operation of converting the fused feature into a query matrix, and the front view features of the nth frame before the current frame are converted into a key matrix and a value matrix, which are processed by a cross-attention mechanism to obtain a fused feature is repeated until the front view features of all historical frames are processed to obtain a fused feature map.

[0032] In the step S1, the preset number of frames is no less than two frames.

[0033] The present invention also provides a three-dimensional lane line generation system integrating contextual semantic information, comprising:

[0034] Image acquisition module: used to acquire image data of a preset number of frames and mark lane lines;

[0035] The image data includes a current frame and a historical frame;

[0036] Coordinate transformation module: used to construct a ground coordinate system with the camera as the reference and the intersection of a vertical line extending toward the ground and perpendicular to the ground plane and the ground plane as the coordinate origin;

[0037] Based on the ground coordinate system, the three-dimensional coordinate points of the ground of the current frame image and the historical frame image are obtained, and after three-dimensional anchor sampling, the three-dimensional feature anchor point position of the current frame and the three-dimensional feature anchor point position of the historical frame are obtained;

[0038] Based on the camera external parameters of the current frame image and the historical frame images, the position change of each anchor point in the current frame is calculated;

[0039] Based on the position change, the three-dimensional feature anchor point position of the current frame is transformed into the ground coordinate system of the historical frame through the first coordinate transformation to obtain the transformed anchor point position;

[0040] The camera coordinate system is constructed with the camera as the coordinate origin. Based on the camera intrinsic parameters, the transformed anchor point position is projected to the camera coordinate system after the second coordinate transformation to obtain the two-dimensional feature point position of the current frame;

[0041] Based on the two-dimensional feature point positions of the current frame, the front view features of the current frame are obtained;

[0042] The 3D feature anchor point position of the historical frame is projected to the camera coordinate system after the second coordinate transformation to obtain the front view feature of the historical frame;

[0043] Feature fusion module: It is used to process the front view features of the current frame and the front view features of the historical frames through a cross-attention mechanism to obtain a fused feature map;

[0044] 3D lane line generation module: used to fuse feature maps for classification and regression processing to generate 3D lane lines.

[0045] Beneficial effect: The present invention obtains relatively dense lane line data of multiple frames of images. Based on the projection-sampling strategy, the first coordinate transformation is used to realize the transformation of the anchor point to be transformed from the ground coordinate system of the current frame to the ground coordinate system of the historical frame. Then, the second coordinate transformation is used to transform the anchor point in the ground coordinate system to the front view. Combined with the information of the historical frame, the coordinate transformation of different frames can more accurately match the lane line. Then, the cross-attention mechanism is used to complete the temporal context fusion. The fused feature map can integrate the information of different time frames, provide rich context information, better understand the lane line structure and morphology of the current frame, and enhance the accuracy of three-dimensional lane line generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 A schematic flow chart of a method for generating three-dimensional lane lines by integrating contextual semantic information is provided in an embodiment.

[0047] Figure 2 Schematic diagram of the ground coordinate system and the camera coordinate system.

[0048] Figure 3 Schematic diagram of the effects of the Anchor3DLane method (a) and the method of the present application (b) on road condition 1, where the red line represents the three-dimensional lane line generated by the corresponding experimental method, and the blue line represents the actual position of the lane line. The three figures from left to right represent: front view result, inverse perspective transformation view result, and three-dimensional space result.

[0049] Figure 4 Schematic diagram of the effects of the Anchor3DLane method (a) and the method of the present application (b) on road condition 2, where the red line represents the three-dimensional lane line generated by the corresponding experimental method, and the blue line represents the actual location of the lane line. The three figures from left to right represent: front view result, inverse perspective transformation view result, and three-dimensional space result.

[0050] Figure 5Schematic diagram of the effects of the Gen-LaneNet method (a) and the method of the present application (b) on road condition three, where the red line represents the three-dimensional lane line generated by the corresponding experimental method, and the blue line represents the actual location of the lane line. The three figures from left to right respectively represent: front view result, inverse perspective transformation view result, and three-dimensional space result.

[0051] Figure 6 Schematic diagram of the effects of the Gen-LaneNet method (a) and the method of the present application (b) on road condition four, where the red line represents the three-dimensional lane line generated by the corresponding experimental method, the blue line represents the actual location of the lane line, and the three figures from left to right represent: front view result, inverse perspective transformation view result, and three-dimensional space result.

[0052] Figure 7 Schematic diagram of the effects of the 3D-LaneNet method (a) and the method of the present application (b) on road condition five, where the red line represents the three-dimensional lane line generated by the corresponding experimental method, the blue line represents the actual location of the lane line, and the three figures from left to right represent: front view result, inverse perspective transformation view result, and three-dimensional space result.

[0053] Figure 8 Schematic diagram of the effects of the 3D-LaneNet method (a) and the method of the present application (b) on road condition six, where the red line represents the three-dimensional lane line generated by the corresponding experimental method, the blue line represents the actual location of the lane line, and the three figures from left to right represent: front view result, inverse perspective transformation view result, and three-dimensional space result. DETAILED DESCRIPTION

[0054] The following examples are intended to illustrate the present invention rather than to further limit the present invention.

[0055] The present invention provides a method for generating three-dimensional lane lines by integrating contextual semantic information. Figure 1 As shown, the following steps are included:

[0056] S1: Obtain image data of a preset number of frames and mark lane lines;

[0057] The image data includes a current frame and a historical frame.

[0058] First of all, the acquisition of image data comes from the capture of the camera (camcorder), such as selecting the data captured by the camera at the front of the vehicle based on actual driving habits and conditions.

[0059] Then, the annotations, including the location information of lane lines, can be represented in the form of polygons or keypoints.

[0060] In addition, the preset number of frames is no less than two frames, such as 2 to 5 frames. Considering the computational burden and combining the real-time requirements of actual applications, 2 frames is preferred.

[0061] S2: With the camera as the reference, the intersection of the vertical line extending toward the ground and perpendicular to the ground plane and the ground plane is taken as the coordinate origin to construct the ground coordinate system;

[0062] Based on the ground coordinate system, the three-dimensional coordinate points of the ground of the current frame image and the historical frame image are obtained, and after three-dimensional anchor sampling, the three-dimensional feature anchor point position of the current frame and the three-dimensional feature anchor point position of the historical frame are obtained;

[0063] Based on the camera external parameters of the current frame image and the historical frame images, the position change of each anchor point in the current frame is calculated;

[0064] Based on the position change, the three-dimensional feature anchor point position of the current frame is transformed into the ground coordinate system of the historical frame through the first coordinate transformation to obtain the transformed anchor point position;

[0065] The camera coordinate system is constructed with the camera as the coordinate origin. Based on the camera intrinsic parameters, the transformed anchor point position is projected to the camera coordinate system after the second coordinate transformation to obtain the two-dimensional feature point position of the current frame;

[0066] Based on the two-dimensional feature point positions of the current frame, the front view features of the current frame are obtained;

[0067] The 3D feature anchor point position of the historical frame is projected to the camera coordinate system after the second coordinate transformation to obtain the front view feature of the historical frame.

[0068] Figure 2 The figure shows the two types of coordinate systems in step S2: the ground coordinate system and the camera coordinate system.

[0069] Regarding the ground coordinate system, its coordinate origin is the intersection of the vertical line extending toward the ground and perpendicular to the ground plane with the camera as the reference, such as Figure 2 The Xg-Yg-Zg coordinate system with Og as the origin is constructed in p 1 、p 2 、p 3 、p 4 Represents the three-dimensional coordinate point on the ground.

[0070] In addition, regarding the 3D anchor sampling, in the ground coordinate system, the 3D anchor is a ray starting from (X0, 0, 0) with a pitch angle θ and a yaw angle φ. Multiple rays constitute the total 3D anchor. N points of each anchor are sampled by the y coordinates of the same fixed distance, through represents the jth 3D anchor, where a 3D anchor q j The kth point of express. Figure 2 middle q 1 、q 2 、q 3 、q 4 Represents a 3D feature anchor point.

[0071] Regarding the camera coordinate system, its coordinate origin is the camera, such as Figure 2 The Xc-Yc-Zc coordinate system constructed with Oc as the origin. q´ 1 、q´ 2 、q´ 3 、q´ 4 Respectively q 1 、q 2 、q 3 、q 4 2D feature points projected to the camera coordinate system.

[0072] As can be seen from the above, the anchor points obtained by 3D anchor sampling and the front view captured by the camera are not in the same coordinate system, and the generation of 3D lane lines needs to be based on the front view. Therefore, coordinate transformation is required.

[0073] The position change is actually the offset of the same point in two frames in a coordinate system, which is closely related to the camera's external parameters. The camera's external parameters include rotation matrices and displacement vectors, which are related to the pose transformation. Preferably, based on the camera's external parameters of the current frame image and the historical frame image, the position change of each anchor point in the current frame is calculated, specifically:

[0074] According to the formula: , get the position change of each anchor point in the current frame;

[0075] In the formula, ( Tx , Ty , Tz ) represents the position change of each anchor point in the current frame; R2 is the rotation matrix of the historical frame; t1 and t2 are the displacement vectors of the current frame and the historical frame respectively.

[0076] Preferably, the three-dimensional feature anchor point position of the current frame is subjected to a first coordinate transformation based on the position change amount to obtain a transformed anchor point position, specifically:

[0077] According to the formula: , get the transformed anchor point position;

[0078] In the formula, x ´、 y ´、 z ' is the anchor point after transformation x axis, y axis, z The coordinate values ​​of the axis; x , y , z is the 3D feature anchor point of the current frame x axis, y axis, z The coordinate values ​​of the axis; Tx , Ty , Tz Respectively represent x , y , z exist x direction, y direction, z The change in position of the direction; is the feature transformation matrix.

[0079] The present invention realizes the transformation of the anchor point to be transformed from the ground coordinate system of the current frame to the ground coordinate system of the historical frame through the first coordinate transformation.

[0080] In order to realize the operation processing on the front view, the anchor point needs to be projected onto the front view. Preferably, the camera coordinate system is constructed with the camera as the coordinate origin, and based on the camera intrinsic parameters, the transformed anchor point position is projected onto the camera coordinate system after the second coordinate transformation to obtain the two-dimensional feature point position of the current frame, specifically:

[0081] According to the formula: , get the two-dimensional feature point position of the current frame;

[0082] In the formula, u k , v k is the two-dimensional feature point of the current frame u , v The coordinate values ​​of the axis; Anchor k The vertical distance to the plane where the camera is located, and the plane where the camera is located is perpendicular to the direction of camera movement; K is the camera internal parameter; Represents the transformation matrix from the ground coordinate system to the camera coordinate system; , , Anchor k exist x axis, y axis, z The coordinate value after axis transformation.

[0083] The present invention realizes the transformation of the anchor point in the ground coordinate system into the front view through the second coordinate transformation, and finally obtains the front view features of the current frame and the front view features of the historical frame.

[0084] The present invention is based on a projection-sampling strategy, which transforms the feature anchor point of the current frame into the front view of the historical frame through the first coordinate transformation and the second coordinate transformation in sequence, and combines the information of the historical frame so that the coordinate transformation of different frames can more accurately match the lane line, which facilitates the accuracy and completeness of the three-dimensional lane line generation, and can also effectively deal with the influence of camera shake or environmental changes to ensure reliability.

[0085] S3: The front view features of the current frame and the front view features of the historical frames are processed by the cross attention mechanism to obtain a fused feature map. The operation is as follows:

[0086] The front view features of the current frame are converted into a query matrix, and the front view features of the historical frames are converted into a key matrix and a value matrix. The dot product of the query matrix and the key matrix is ​​used to calculate the attention weight. After the dot product operation of the attention weight and the value matrix, the sum is obtained to obtain a fused feature map.

[0087] Preferably, for a straight lane line, the preset number of frames is two frames, the front view feature of the current frame is converted into a query matrix, and the front view feature of a certain frame in the history frame is converted into a key matrix and a value matrix, and the fused features obtained by the cross attention mechanism are recorded as a fused feature map;

[0088] For curved lane lines, if the preset number of frames is more than two frames, the front view features of the current frame are converted into a query matrix, and the front view features of the first frame before the current frame are converted into a key matrix and a value matrix, which are processed by a cross-attention mechanism to obtain a first fused feature; the first fused feature is converted into a query matrix, and the front view features of the second frame before the current frame are converted into a key matrix and a value matrix, which are processed by a cross-attention mechanism to obtain a second fused feature; the operation of converting the fused feature into a query matrix, and the front view features of the nth frame before the current frame are converted into a key matrix and a value matrix, which are processed by a cross-attention mechanism is repeated until the front view features of all historical frames are processed to obtain a fused feature map.

[0089] For example, take the three-frame front view features as an example. These three frames are converted into vector representations, named e1, e2, and e3 respectively. The process of obtaining the fused feature map is:

[0090] Take e3 as the query matrix, e2 as the key matrix and value matrix, and the result after processing by the cross attention mechanism as the new query matrix, and e1 as the key matrix and value matrix, and the result after processing by the cross attention mechanism as the fused feature map.

[0091] For the lane line data with relatively dense multi-frame images acquired, the present invention utilizes the similarity between frames to fuse the features of multiple inputs, and completes the temporal context fusion through cross-attention mechanism processing. The fused feature map can integrate the information of different time frames, provide rich contextual information, enhance the feature expression ability of the lane line, better understand the lane line structure and form of the current frame, and better predict the future lane line shape and position to generate three-dimensional lane lines.

[0092] S4: Fusion feature maps are used for classification and regression processing to generate three-dimensional lane lines.

[0093] The present invention obtains relatively dense lane line data of multiple frames of images. Based on the projection-sampling strategy, the first coordinate transformation is first used to transform the anchor point to be transformed from the ground coordinate system of the current frame to the ground coordinate system of the historical frame. Then, the second coordinate transformation is used to transform the anchor point in the ground coordinate system to the front view. Combined with the information of the historical frames, the coordinate transformations of different frames can more accurately match the lane lines. Then, the cross-attention mechanism is used to complete the temporal context fusion. The fused feature map can integrate the information of different time frames, provide rich context information, better understand the lane line structure and morphology of the current frame, and enhance the accuracy of three-dimensional lane line generation.

[0094] The three-dimensional lane line generation method provided by the present invention can better cope with changes in complex environments while maintaining good real-time performance. It can adapt to complex traffic scenarios and different weather conditions, provide comprehensive environmental perception for vehicles, thereby optimizing driving experience and reducing accident risks. It is suitable for autonomous driving and assisted driving systems, and shows broad application prospects in the fields of intelligent transportation and autonomous driving.

[0095] The present invention also provides a three-dimensional lane line generation system integrating contextual semantic information, comprising:

[0096] Image acquisition module: used to acquire image data of a preset number of frames and mark lane lines;

[0097] The image data includes a current frame and a historical frame;

[0098] Coordinate transformation module: used to construct a ground coordinate system with the camera as the reference and the intersection of a vertical line extending toward the ground and perpendicular to the ground plane and the ground plane as the coordinate origin;

[0099] Based on the ground coordinate system, the three-dimensional coordinate points of the ground of the current frame image and the historical frame image are obtained, and after three-dimensional anchor sampling, the three-dimensional feature anchor point position of the current frame and the three-dimensional feature anchor point position of the historical frame are obtained;

[0100] Based on the camera external parameters of the current frame image and the historical frame images, the position change of each anchor point in the current frame is calculated;

[0101] Based on the position change, the three-dimensional feature anchor point position of the current frame is transformed into the ground coordinate system of the historical frame through the first coordinate transformation to obtain the transformed anchor point position;

[0102] The camera coordinate system is constructed with the camera as the coordinate origin. Based on the camera intrinsic parameters, the transformed anchor point position is projected to the camera coordinate system after the second coordinate transformation to obtain the two-dimensional feature point position of the current frame;

[0103] Based on the two-dimensional feature point positions of the current frame, the front view features of the current frame are obtained;

[0104] The 3D feature anchor point position of the historical frame is projected to the camera coordinate system after the second coordinate transformation to obtain the front view feature of the historical frame;

[0105] Feature fusion module: It is used to process the front view features of the current frame and the front view features of the historical frames through a cross-attention mechanism to obtain a fused feature map;

[0106] 3D lane line generation module: used to fuse feature maps for classification and regression processing to generate 3D lane lines.

[0107] Experimental Evaluation

[0108] In order to better reflect the generation effect of the method of this application, three comparison methods are used for comparison, namely:

[0109] The Anchor3DLane method realizes the structured modeling of 3D lane lines by designing lane line anchor points in 3D space. The 3D anchor points are projected onto the front view feature (FV) image using the internal and external parameters of the camera to perform 3D anchor feature sampling. Anchor3DLane processes lane detection at a certain moment. The method of this application enhances the detection effect at the current moment by fusion of multiple frames in time series to generate accurately positioned 3D lane lines.

[0110] Depend on Figure 3 , Figure 4 It can be seen that Anchor3DLane misses detection when the road curvature changes, while the method of this application has excellent generation effect. Anchor3DLane has a larger error at a longer distance in curved sections, while the result of the method of this application is close to the result of the actual lane line. Therefore, compared with the Anchor3DLane method, the method of this application has higher robustness and accuracy when dealing with complex road environments, especially curves, uphill and downhill situations.

[0111] The Gen-LaneNet method extracts 3D lane information from a single image. This method combines image encoding, feature space transformation, and 3D lane line generation in a unified framework. However, Gen-LaneNet does not work well at long distances, while the method in this application can combine temporal information.

[0112] Depend on Figure 5 It can be seen that the Gen-LaneNet method is prone to missed detection at night, for example, only 4 lane lines are generated out of 5 lane lines, while the method of this application has an excellent generation effect. Figure 6 It can be seen that in the inverse perspective transformation view angle and three-dimensional space results, Gen-LaneNet has a larger error at a longer distance in darker turning sections, while the method of this application can reduce this error. Therefore, compared with the Gen-LaneNet method, the method of this application has higher accuracy in processing night road environments.

[0113] The 3D-LaneNet method is an end-to-end monocular 3D lane detector that converts the features in the front view feature (FV) into a bird's-eye view (BEV) representation through inverse perspective mapping (IPM), and then regresses the anchor offset of the lane in the BEV space. However, 3D-LaneNet does not work well in environments such as uphill and downhill, while the method in this application can adapt to more complex environments.

[0114] Depend on Figure 7 , Figure 8 It can be seen that 3D-LaneNet is prone to missed detection when going uphill or downhill, while the method of this application has higher accuracy. 3D-LaneNet is prone to missed detection when on a sloped turning section, while the method of this application is relatively accurate and achieves a lower error. Therefore, compared with the 3DLaneNet method, the method of this application has higher accuracy in dealing with changes in road curvature, especially uphill or downhill and slope sections.

[0115] Therefore, the method of this application can provide more stable and accurate lane line generation results in various complex road environments. In particular, this method shows significant advantages in special situations such as curves, uphill and downhill, and low light at night. Compared with existing methods such as Anchor3DLane, Gen-LaneNet and 3D-LaneNet, the method of this application can effectively overcome the problems of missed detection, misjudgment and large errors that may occur in these methods under specific environments.

Claims

1. A three-dimensional lane line generation method integrating contextual semantic information, characterized in that: The following steps are involved: S1: Obtain image data of a preset number of frames and mark lane lines; The image data includes a current frame and a historical frame; S2: With the camera as the reference, the intersection of the vertical line extending toward the ground and perpendicular to the ground plane and the ground plane is taken as the coordinate origin to construct the ground coordinate system; Based on the ground coordinate system, the three-dimensional coordinate points of the ground of the current frame image and the historical frame image are obtained, and after three-dimensional anchor sampling, the three-dimensional feature anchor point position of the current frame and the three-dimensional feature anchor point position of the historical frame are obtained; Based on the camera external parameters of the current frame image and the historical frame images, the position change of each anchor point in the current frame is calculated; Based on the position change, the three-dimensional feature anchor point position of the current frame is transformed into the ground coordinate system of the historical frame through the first coordinate transformation to obtain the transformed anchor point position; The camera coordinate system is constructed with the camera as the coordinate origin. Based on the camera intrinsic parameters, the transformed anchor point position is projected to the camera coordinate system after the second coordinate transformation to obtain the two-dimensional feature point position of the current frame; Based on the two-dimensional feature point positions of the current frame, the front view features of the current frame are obtained; The 3D feature anchor point position of the historical frame is projected to the camera coordinate system after the second coordinate transformation to obtain the front view feature of the historical frame; S3: The front view features of the current frame and the front view features of the historical frames are processed by the cross attention mechanism to obtain a fused feature map; S4: Fusion feature maps are used for classification and regression processing to generate three-dimensional lane lines.

2. The method for generating three-dimensional lane lines by integrating contextual semantic information according to claim 1, characterized in that: In step S2, based on the position change, the three-dimensional feature anchor point position of the current frame is transformed by a first coordinate transformation to obtain a transformed anchor point position, which is specifically: According to the formula: , get the transformed anchor point position; In the formula, x ´、 y ´、 z ' is the anchor point after transformation x axis, y axis, z The coordinate values ​​of the axis; x , y , z is the 3D feature anchor point of the current frame x axis, y axis, z The coordinate values ​​of the axis; Tx , Ty , Tz Respectively represent x , y , z exist x direction, y direction, z The change in position of the direction; is the feature transformation matrix.

3. The method for generating three-dimensional lane lines by integrating contextual semantic information according to claim 1, characterized in that: In step S2, a camera coordinate system is constructed with the camera as the coordinate origin. Based on the camera intrinsic parameters, the transformed anchor point position is projected to the camera coordinate system after a second coordinate transformation to obtain the two-dimensional feature point position of the current frame, specifically: According to the formula: , get the two-dimensional feature point position of the current frame; In the formula, u k , v k is the two-dimensional feature point of the current frame u , v The coordinate values ​​of the axis; Anchor k The vertical distance to the plane where the camera is located, and the plane where the camera is located is perpendicular to the direction of camera movement; K is the camera internal parameter; Represents the transformation matrix from the ground coordinate system to the camera coordinate system; , , Anchor k exist x axis, y axis, z The coordinate value after axis transformation.

4. The method for generating three-dimensional lane lines by integrating contextual semantic information according to claim 1, characterized in that: In step S2, based on the camera extrinsic parameters of the current frame image and the historical frame image, the position change of each anchor point in the current frame is calculated, specifically: According to the formula: , get the position change of each anchor point in the current frame; In the formula, ( Tx , Ty , Tz ) represents the position change of each anchor point in the current frame; R2 is the rotation matrix of the historical frame; t1 and t2 are the displacement vectors of the current frame and the historical frame respectively.

5. The method for generating three-dimensional lane lines by integrating contextual semantic information according to claim 1, characterized in that: In step S3, the front view features of the current frame and the front view features of the historical frames are processed by a cross attention mechanism to obtain a fused feature map, and the operation is as follows: The front view features of the current frame are converted into a query matrix, and the front view features of the historical frames are converted into a key matrix and a value matrix. The dot product of the query matrix and the key matrix is ​​used to calculate the attention weight. After the dot product operation of the attention weight and the value matrix, the sum is obtained to obtain a fused feature map.

6. The method for generating three-dimensional lane lines by integrating contextual semantic information according to claim 5, characterized in that: The front view features of the current frame and the front view features of the historical frames are processed by the cross attention mechanism to obtain a fused feature map, which is: For straight lane lines, the preset number of frames is two. The front view features of the current frame are converted into a query matrix, and the front view features of a certain frame in the history frame are converted into a key matrix and a value matrix. After being processed by the cross attention mechanism, the obtained fusion features are recorded as a fusion feature map. For curved lane lines, if the preset number of frames is more than two, the front view features of the current frame are converted into a query matrix, and the front view features of the first frame before the current frame are converted into a key matrix and a value matrix, which are processed by a cross attention mechanism to obtain the first fusion feature; The first fused feature is converted into a query matrix, and the front view feature of the second frame before the current frame is converted into a key matrix and a value matrix, which are processed by a cross attention mechanism to obtain the second fused feature; the fused feature is repeatedly converted into a query matrix, and the front view feature of the nth frame before the current frame is converted into a key matrix and a value matrix, which are processed by a cross attention mechanism to obtain the fused feature operation until all the front view features of the historical frames are processed to obtain a fused feature map.

7. The method for generating three-dimensional lane lines by integrating contextual semantic information according to claim 1, characterized in that: In the step S1, the preset number of frames is no less than two frames.

8. A three-dimensional lane line generation system integrating contextual semantic information, characterized in that: include: Image acquisition module: used to acquire image data of a preset number of frames and mark lane lines; The image data includes a current frame and a historical frame; Coordinate transformation module: used to construct a ground coordinate system with the camera as the reference and the intersection of a vertical line extending toward the ground and perpendicular to the ground plane and the ground plane as the coordinate origin; Based on the ground coordinate system, the three-dimensional coordinate points of the ground of the current frame image and the historical frame image are obtained, and after three-dimensional anchor sampling, the three-dimensional feature anchor point position of the current frame and the three-dimensional feature anchor point position of the historical frame are obtained; Based on the camera external parameters of the current frame image and the historical frame images, the position change of each anchor point in the current frame is calculated; Based on the position change, the three-dimensional feature anchor point position of the current frame is transformed into the ground coordinate system of the historical frame through the first coordinate transformation to obtain the transformed anchor point position; The camera coordinate system is constructed with the camera as the coordinate origin. Based on the camera intrinsic parameters, the transformed anchor point position is projected to the camera coordinate system after the second coordinate transformation to obtain the two-dimensional feature point position of the current frame; Based on the two-dimensional feature point positions of the current frame, the front view features of the current frame are obtained; The 3D feature anchor point position of the historical frame is projected to the camera coordinate system after the second coordinate transformation to obtain the front view feature of the historical frame; Feature fusion module: It is used to process the front view features of the current frame and the front view features of the historical frames through a cross-attention mechanism to obtain a fused feature map; 3D lane line generation module: used to fuse feature maps for classification and regression processing to generate 3D lane lines.

Images