Lane center line and topological relation generation system and method, electronic equipment and medium

Abstract

The invention relates to a lane center line and topological relation generation system and method, electronic equipment and a medium, and the system comprises a BEV spatial feature extraction module which converts vehicle panoramic video data into a first BEV feature map, converts navigation map data into a second BEV feature map, and fuses the first BEV feature map and the second BEV feature map into a feature map; the feature information interaction module is used for processing the feature map and outputting a query matrix fused with feature map information; calculating topological relation fusion among different lane center lines according to the query matrix, and outputting the query matrix fused with the topological relation of the lane center lines; and the center line and topological relation prediction module is used for predicting the three-dimensional coordinates of the composition nodes of the center line of each lane and the confidence coefficient of the center line of the lane on the basis of the query matrix, and predicting the topological relation of the two center lines on the basis of the query matrix fused with the topological relation of the center lines of the lane. According to the invention, the generation of the lane center line and the topological relation thereof based on the visual data and the map navigation data is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of assisted driving lane environment perception, and specifically relates to a lane centerline and topological relationship generation system, method, electronic equipment and medium. Background Art

[0002] Patent document CN109186617A discloses a method, system and memory for automatically generating lane-level topological relationships based on visual crowdsourcing data, including: step 1, generating a lane centerline based on lane edge data obtained by a crowdsourcing camera; step 2, generating a passable path line in a lane-level intersection by clustering and fitting the trajectory according to the driving direction and trajectory point heading angle when there is no lane edge data in the intersection, and determining the first and last points of the passable path line; step 3, collecting the lane edges on both sides of the same section along the direction of travel based on the plane position relationship; step 4, splicing the lane centerline and the endpoints of the trajectory line in the intersection to construct a complete lane-level road network connection relationship based on the principle of the closest plane position distance. Among them, the ordered point sequence of step 1 is affected by the graphics processing generation algorithm. The clustering effect of step 2 is affected by the number of cluster clusters and the selected threshold. These two parameters should be manually selected with different values ​​for different types of intersections, so their generalization is general. Step 3 does not explain how the centerline should be connected at the place where the number of lanes changes. In addition, this method does not clearly define the demarcation criteria for lanes and intersections, and it is unclear when to use image data to generate ordered point sequences and when to use trajectory points and heading angle data. In general, this method uses artificially designed features to calculate lane lines, lane centerlines, intersection centerlines, and topological relationships, resulting in poor generalization. Secondly, no clear definition and calculation method is given for lane and intersection demarcations and lane changes, resulting in poor robustness and inability to handle the above complex scenarios.

[0003] Patent document CN115937825A discloses a robust lane line generation method and device under BEV with online pitch angle estimation. The method includes: based on lane line information and other information, completing pixel-level dense segmentation of the front-view monocular image to obtain the corresponding image mask information; extracting multiple groups of lane lines that satisfy parallel relationships in the two-dimensional image plane according to the image mask information; combining the unknown pitch angle to construct an external parameter matrix, back-projecting the endpoints of the parallel lane lines on the image plane to the bird's-eye view BEV, and constructing a cost function for the unknown pitch angle according to the lane line parallel prior information; given the resolution and size information, constructing a grid interest region under the bird's-eye view (BEV), substituting it into the solution to obtain the unknown pitch angle, and generating lane lines under the bird's-eye view BEV in combination with the image mask information, thereby more effectively realizing the detection of lane lines. Among them, the lane line recognition and generation tasks are mainly performed based on the front-view image coordinate system. After the lane line is generated (represented in the form of a multi-segment line), it is projected to the BEV coordinate system through the internal and external parameters of the camera. The method is generally feasible, but has the following defects: 1) After completing the pixel-level dense segmentation, the threshold needs to be manually specified when determining the lane line target; 2) When using the straight line feature extraction algorithm to segment the shortest pixel-level path of the endpoints on both sides of the lane line, the threshold needs to be manually specified; 3) When estimating the unknown pitch angle α, its basic assumption is the plane section assumption, that is, in the process of reverse perspective, it is assumed that the road surface is horizontal. Therefore, in areas with large ground slopes, the algorithm has low accuracy. In addition, when solving the unknown pitch angle α, the basic assumption of the cost function construction is that the lane lines on both sides of the vehicle are parallel. Therefore, in the transition section where the vehicle enters the curve section from the straight section, the algorithm cannot solve the correct unknown pitch angle α.

[0004] Although the above two methods can generate lane centerlines, lane edges and topological relationships to a certain extent, they rely on a large number of artificially designed features and thresholds, resulting in insufficient generalization and robustness. In addition, the above two methods do not propose corresponding methods for some complex scenarios in assisted driving, such as where the number of lanes changes and the transition from a straight segment to a curved segment.

[0005] Therefore, it is necessary to develop a new lane centerline and topological relationship generation system, method, electronic device and medium. Summary of the invention

[0006] The purpose of the present invention is to provide a lane centerline and topological relationship generation system, method, electronic device and medium, which can realize the generation of lane centerlines and their topological relationships based on visual data and navigation map data, and can be applicable to the transition section from the number of lanes to the straight section to the curved section.

[0007] In a first aspect, the lane centerline and topological relationship generation system of the present invention includes: A BEV spatial feature extraction module is used to convert the vehicle surround video data into a first BEV feature map in a vehicle body coordinate system through a feature extraction network, convert the navigation map data into a local coordinate system according to the vehicle posture information and the BEV feature space range, and perform BEV encoding processing to obtain a second BEV feature map, and perform feature map fusion on the first BEV feature map and the second BEV feature map to obtain a fused feature map; The feature information interaction module uses a deformable cross self-attention mechanism to process the feature map at each moment, and outputs a query matrix that integrates the feature map information, where the query matrix is ​​a plurality of feature vectors that represent the lane centerline information; and calculates the topological relationship fusion between different lane centerlines through the GCN network based on the query matrix that integrates the feature map information, outputs a query matrix that integrates the lane centerline topological relationship, and uses the query matrix that integrates the lane centerline topological relationship as the query matrix for the next feature information interaction, achieving full information interaction through multiple rounds of iterations; The centerline and topological relationship prediction module predicts the three-dimensional coordinates of the nodes constituting the centerline of each lane and the confidence level of the centerline of the lane based on a query matrix that incorporates the topological relationship of the centerlines of the lanes; and predicts the topological relationship between two centerlines based on a query matrix that incorporates the topological relationship of the centerlines of the lanes.

[0008] Optionally, the image feature map generation submodule uses a convolutional neural network to extract features from the image taken by each camera at each moment, and performs multi-scale fusion to generate a feature map; through the inverse perspective method and in combination with the internal and external parameters of the camera, the feature maps of multiple cameras at the same moment are converted to the BEV space, and are fused and output as the first BEV feature map in the ego-vehicle coordinate system, that is, a first BEV feature map based on the ego-vehicle coordinate system is generated at each time step; The navigation map encoding submodule converts the navigation map data into a local coordinate system according to the self-vehicle posture information and the BEV feature space range, and performs BEV encoding processing to obtain a second BEV feature map; The data fusion submodule stacks the first BEV feature map and the second BEV feature map in the direction of the number of channels, and then projects them into the target number of channels through the perceptron to form a fused feature map.

[0009] Optionally, the feature information interaction module includes: The feature information interaction submodule uses a deformable cross self-attention mechanism to process the feature map at each moment and outputs a query matrix that integrates the feature map information. The query matrix is ​​a number of feature vectors that represent the lane centerline information. The feature information propagation submodule receives the query matrix that integrates the feature map information output by the feature information interaction submodule, calculates the topological relationship fusion between different lane centerlines through the GCN network, outputs the query matrix that integrates the topological relationship of the lane centerlines, and uses the query matrix that integrates the topological relationship of the lane centerlines as the query matrix for the next feature information interaction, realizing full information interaction through multiple rounds of iterations.

[0010] Optionally, the centerline and topological relationship prediction module includes: The first multi-layer perceptron predicts the coordinates of each lane centerline node based on the query matrix output by the feature information interaction module that incorporates the lane centerline topological relationship; The second multi-layer perceptron predicts the confidence of each lane centerline based on the query matrix output by the feature information interaction module that integrates the topological relationship of the lane centerlines; The third multi-layer perceptron is used to reduce the dimension of the query matrix output by the feature information interaction module that integrates the topological relationship of the lane centerline; The fourth multi-layer perceptron is used to stack the features of any two lane center lines in the query matrix after dimension reduction in the channel direction; The fifth multi-layer perceptron predicts the topological relationship between the center lines of the two lanes based on the stacking results.

[0011] In a second aspect, a method for generating a lane centerline and a topological relationship according to the present invention is characterized in that it comprises the following steps: S1: Acquire vehicle surround video data, and convert the vehicle surround video data into a first BEV feature map in a vehicle body coordinate system through a feature extraction network; acquire navigation map data, convert the navigation map data into a local coordinate system according to the vehicle posture information and the BEV feature space range, and perform BEV encoding processing to obtain a second BEV feature map, and perform feature map fusion on the first BEV feature map and the second BEV feature map to obtain a fused feature map; S2: Use a deformable cross self-attention mechanism to process the feature map at each moment, and output a query matrix that integrates the feature map information, where the query matrix is ​​a plurality of feature vectors that represent the lane centerline information; and calculate the topological relationship fusion between different lane centerlines through the GCN network based on the query matrix that integrates the feature map information, output a query matrix that integrates the lane centerline topological relationship, and use the query matrix that integrates the lane centerline topological relationship as the query matrix for the next feature information interaction, and achieve full information interaction through multiple rounds of iterations; S3: Predict the three-dimensional coordinates of the constituent nodes of each lane centerline and the confidence level of the lane centerline based on a query matrix that incorporates the topological relationship of the lane centerlines; and predict the topological relationship of two centerlines based on a query matrix that incorporates the topological relationship of the lane centerlines.

[0012] Optionally, the S1 is specifically: For the images taken by each camera at each moment, a convolutional neural network is used to extract features and perform multi-scale fusion to generate a feature map. By using the inverse perspective method and combining the internal and external parameters of the camera, the feature maps of multiple cameras at the same moment are converted to the BEV space and fused to output the first BEV feature map in the ego-vehicle coordinate system. That is, a first BEV feature map based on the ego-vehicle coordinate system is generated at each time step. The navigation map data is converted into a local coordinate system according to the self-vehicle posture information and the BEV feature space range, and BEV encoding is performed to obtain a second BEV feature map; The first BEV feature map and the second BEV feature map are stacked in the direction of the number of channels, and then projected into the target number of channels through the perceptron to form a fused feature map.

[0013] Optionally, the S3 includes: The coordinates of each lane centerline node are predicted based on the query matrix that incorporates the lane centerline topological relationship; Predict the confidence of each lane centerline based on the query matrix that incorporates the topological relationship of the lane centerlines; Reduce the dimension of the query matrix that incorporates the lane centerline topology; Stack the reduced-dimensional features of any two lane centerlines in the query matrix in the channel direction; Based on the stacking results, predict the topological relationship between the center lines of the two lanes.

[0014] In a third aspect, an electronic device described in the present invention includes a memory and a controller, wherein the memory stores a computer-readable program, and when the computer-readable program is called by the controller, it can execute the steps of the method for generating lane center lines and topological relationships as described in the present invention.

[0015] In a fourth aspect, a medium described in the present invention is characterized in that: a computer-readable program is stored therein, and when the computer-readable program is called by a controller, it can execute the steps of the lane centerline and topological relationship generation method described in the present invention.

[0016] The present invention has the following advantages: The present invention directly generates lane centerlines through a deep learning model based on vehicle surround video data and navigation map data, rather than through indirect data such as trajectory points or lane edges, thereby reducing errors and achieving end-to-end instant mapping. In addition, the present invention also generates topological connection relationships based on lane centerlines, and the results can be directly used for downstream assisted driving planning and control. The present invention automatically formulates rules through training data to achieve end-to-end lane centerline generation and its topological relationship method from the perspective of BEV. The present invention can improve generalization and robustness, and can handle complex scenes such as changes in the number of lanes and transition sections from straight sections to curved sections. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 is a functional block diagram of this embodiment; Figure 2 is a principle block diagram of the BEV spatial feature extraction module in this embodiment; Figure 3 is a principle block diagram of the feature information interaction module in this embodiment; Figure 4 Schematic diagram of the centerline and topological relationship prediction module in this embodiment; Figure 5 is a flow chart of this embodiment; Figure 6 : is a workflow diagram of the BEV space feature extraction module in this embodiment; Figure 7 is a workflow diagram of the feature information interaction module in this embodiment; Figure 8 : is a workflow diagram of the centerline and topological relationship prediction module in this embodiment; In the figure: 1. BEV spatial feature extraction module, 11. Image feature map generation submodule, 12. Navigation map encoding submodule, 13. Data fusion submodule, 2. Feature information interaction module, 21. Feature information interaction submodule, 22. Feature information propagation submodule, 3. Centerline and topological relationship prediction module, 31. First multi-layer perceptron, 32. Second multi-layer perceptron, 33. Third multi-layer perceptron, 34. Fourth multi-layer perceptron, 35. Fifth multi-layer perceptron. DETAILED DESCRIPTION

[0019] The embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be construed as being limited to the embodiments described herein. Instead, these embodiments are provided to provide a more thorough and complete understanding of the present application. It should be understood that the drawings and embodiments of the present application are only for exemplary purposes and are not intended to limit the scope of protection of the present application.

[0020] like Figure 1 As shown, in this embodiment, a lane centerline and topological relationship generation system includes a BEV space feature extraction module 1, a feature information interaction module 2 and a centerline and topological relationship prediction module 3. Among them, the BEV space feature extraction module 1 is used to convert the vehicle surrounding video data into a first BEV feature map in the vehicle body coordinate system through a feature extraction network, convert the navigation map data to the local coordinate system according to the vehicle posture information and the BEV feature space range, and perform BEV encoding processing to obtain a second BEV feature map, and perform feature map fusion on the first BEV feature map and the second BEV feature map to obtain a fused feature map. The feature information interaction module 2 uses a deformable cross self-attention mechanism to process the feature map at each moment, and outputs a query matrix that fuses the feature map information, wherein the query matrix is ​​a plurality of feature vectors that characterize the lane centerline information; and according to the query matrix that fuses the feature map information, the topological relationship fusion between different lane centerlines is calculated through the GCN network, and the query matrix that fuses the lane centerline topological relationship is output, and the query matrix that fuses the lane centerline topological relationship is used as the query matrix for the next feature information interaction, and the full interaction of information is achieved through multiple rounds of iterations. The centerline and topological relationship prediction module 3 predicts the three-dimensional coordinates of the constituent nodes of each lane centerline and the confidence level of the lane centerline based on a query matrix that integrates the lane centerline topological relationship; and predicts the topological relationship between two centerlines based on a query matrix that integrates the lane centerline topological relationship.

[0021] like Figure 2As shown, in this embodiment, the BEV space feature extraction module 1 includes an image feature map generation submodule 11, a navigation map encoding submodule 12 and a data fusion submodule 13, wherein the image feature map generation submodule 11 uses a convolutional neural network to extract features for the images taken by each camera at each moment, and performs multi-scale fusion to generate a feature map; through the inverse perspective method, combined with the internal and external parameters of the camera, the feature maps of multiple cameras at the same moment are converted to the BEV space, and fused, and output as the first BEV feature map in the self-vehicle coordinate system, that is, a first BEV feature map based on the self-vehicle coordinate system is generated at each time step. The navigation map encoding submodule 12 converts the navigation map data to the local coordinate system according to the self-vehicle posture information and the BEV feature space range, and performs BEV encoding processing to obtain a second BEV feature map. The data fusion submodule 13 stacks the first BEV feature map and the second BEV feature map in the direction of the number of channels, and then projects them into the target number of channels through the perceptron to form a fused feature map.

[0022] like Figure 3 As shown, in this embodiment, the feature information interaction module 2 includes a feature information interaction submodule 21 and a feature information propagation submodule 22, wherein the feature information interaction submodule 21 uses a deformable cross self-attention mechanism to process the feature graph at each moment, and outputs a query matrix that integrates the feature graph information, wherein the query matrix is ​​a plurality of feature vectors that characterize the lane centerline information. The feature information propagation submodule 22 receives the query matrix that integrates the feature graph information output by the feature information interaction submodule 21, calculates the topological relationship fusion between different lane centerlines through the GCN network, outputs the query matrix that integrates the lane centerline topological relationship, and uses the query matrix that integrates the lane centerline topological relationship as the query matrix for the next feature information interaction, and realizes full information interaction through multiple rounds of iterations.

[0023] like Figure 4As shown, in this embodiment, the centerline and topological relationship prediction module 3 includes a first multilayer perceptron 31, a second multilayer perceptron 32, a third multilayer perceptron 33, a fourth multilayer perceptron 34 and a fifth multilayer perceptron 35. Among them: the first multilayer perceptron 31 predicts the coordinates of each lane centerline node based on the query matrix output by the feature information interaction module 2 that integrates the lane centerline topological relationship. The second multilayer perceptron 32 predicts the confidence of each lane centerline based on the query matrix output by the feature information interaction module 2 that integrates the lane centerline topological relationship. The third multilayer perceptron 33 is used to reduce the dimension of the query matrix output by the feature information interaction module 2 that integrates the lane centerline topological relationship. The fourth multilayer perceptron 34 is used to stack the features of any two lane centerlines in the query matrix after dimensionality reduction in the channel direction. The fifth multilayer perceptron 35 predicts the topological relationship between the two lane centerlines based on the stacking result.

[0024] like Figure 5 As shown, in this embodiment, a method for generating a lane centerline and a topological relationship includes the following steps: S1: Acquire vehicle surround video data, and convert the vehicle surround video data into a first BEV feature map in a vehicle body coordinate system through a feature extraction network; acquire navigation map data, convert the navigation map data into a local coordinate system according to the vehicle posture information and the BEV feature space range, and perform BEV encoding processing to obtain a second BEV feature map, and perform feature map fusion on the first BEV feature map and the second BEV feature map to obtain a fused feature map; S2: Use a deformable cross self-attention mechanism to process the feature map at each moment, and output a query matrix that integrates the feature map information, where the query matrix is ​​a plurality of feature vectors that represent the lane centerline information; and calculate the topological relationship fusion between different lane centerlines through the GCN network based on the query matrix that integrates the feature map information, output a query matrix that integrates the lane centerline topological relationship, and use the query matrix that integrates the lane centerline topological relationship as the query matrix for the next feature information interaction, and achieve full information interaction through multiple rounds of iterations; S3: Predict the three-dimensional coordinates of the constituent nodes of each lane centerline and the confidence level of the lane centerline based on a query matrix that incorporates the topological relationship of the lane centerlines; and predict the topological relationship of two centerlines based on a query matrix that incorporates the topological relationship of the lane centerlines.

[0025] In this embodiment, S1 is specifically: For the images taken by each camera at each moment, a convolutional neural network is used to extract features and perform multi-scale fusion to generate a feature map. The feature maps of multiple cameras at the same moment are converted to the BEV space through the inverse perspective method and combined with the internal and external parameters of the camera, and then fused to output the first BEV feature map in the vehicle coordinate system. The navigation map data is converted into a local coordinate system according to the self-vehicle posture information and the BEV feature space range, and BEV encoding is performed to obtain a second BEV feature map; The first BEV feature map and the second BEV feature map are stacked in the direction of the number of channels, and then projected into the target number of channels through the perceptron to form a fused feature map.

[0026] In this embodiment, S3 includes: The coordinates of each lane centerline node are predicted based on the query matrix that incorporates the lane centerline topological relationship; Predict the confidence of each lane centerline based on the query matrix that incorporates the topological relationship of the lane centerlines; Reduce the dimension of the query matrix that incorporates the lane centerline topology; Stack the reduced-dimensional features of any two lane centerlines in the query matrix in the channel direction; Based on the stacking results, predict the topological relationship between the center lines of the two lanes.

[0027] In this embodiment, the original image (i.e., the vehicle surrounding video data) refers to the three-channel image data with time attributes directly collected from the vehicle-side camera, which can be represented as a set of four-dimensional matrices. Considering that there are multiple cameras around the vehicle, if there are N cameras at the same time, there are N such four-dimensional matrices.

[0028] Here is an example of representing the above data in matrix form: Let the dimensions of the original image be (C, H, W), where C represents the number of channels, H represents the height of the image, and W represents the width of the image. Assuming that there are image data at T moments and there are N cameras, it can be expressed as: original image data matrix: X = [X1, X2, ..., XN], where Xi is the image data matrix of the i-th camera at T moments, with dimensions (C, H, W). For example, if there are 2 cameras, the image data captured by each camera at 3 moments is: X1 = [X11, X12, X13], with dimensions (C, H, W), X2 = [X21, X22, X23], with dimensions (C, H, W). Then the original image data matrix can be expressed as: X = [X1, X2], with dimensions (N, T, C, H, W). Among them, N=2 means that there are two cameras, T=3 means that there are image data at three moments, C represents the number of channels, and the present invention takes 3, H represents the height of the image, and W represents the width of the image.

[0029] In this embodiment, the first BEV feature map is the tensor data generated after the original image is processed by the image feature map generation submodule 11, and the first BEV feature map has certain positional features. BEV represents the scene from a bird's-eye view, so the first BEV feature map can be regarded as the positional feature of each point in the scene. The center point of the first BEV feature map is the center of the vehicle, and the range along the vehicle's forward direction and the vertical direction can be flexibly configured according to the capabilities of the camera. At a certain moment, the first BEV feature map is a three-dimensional tensor with a size of Hb × Wb × C, where: Hb represents the size along the vehicle's forward direction, Wb represents the size in the vertical direction of the vehicle's forward direction, and C represents the number of channels, which is 256 in this embodiment. The size of the first BEV feature map depends on the actual BEV range and resolution setting. For example, assuming that the actual range of the BEV is 50 meters before and after the vehicle, 30 meters left and right, and the resolution is set to 0.2 meters, then Hb can be calculated as 100 meters / 0.2 meters = 500, and Wb can be calculated as 60 meters / 0.2 meters = 300. Since the first BEV feature map has a time attribute, the data can be expressed as follows: BEV feature map data: F = [F1, F2, ..., FT], where Fi represents the BEV feature map at the i-th moment, and the dimension is (Hb, Wb, C). For example, if there are BEV feature map data at 3 moments, it can be expressed as: F = [F1, F2, F3], with a dimension of (T, Hb, Wb, C). Among them, T represents the number of moments, Hb represents the size along the direction of the vehicle's forward movement, Wb represents the size in the direction perpendicular to the direction of the vehicle's forward movement, and C represents the number of channels.

[0030] In this embodiment, the query matrix is ​​a learnable matrix with a size of M × dk. M represents the maximum number of lane centerlines that can be generated in the feature map of a certain time step. This parameter is manually specified and is 100 in this embodiment; dk represents the feature dimension of each lane centerline, which is 256 by design. Therefore, the size of the query matrix is ​​100 × 256, where each element is a learnable and updateable value. Each row of the query matrix represents a query vector for a lane centerline.

[0031] In this embodiment, the lane centerline is an entity consisting of 10 ordered nodes and a confidence level. Each ordered node uses three-dimensional relative coordinates (x, y, z) to determine the position on the lane centerline.

[0032] In this embodiment, the topological relationship is: 1) The topological relationship describes the ordered connection between the center lines of two lanes.

[0033] 2) Represented as a two-dimensional matrix of size K × K, where K is the number of lane centerlines.

[0034] 3) The element Eij of the matrix indicates whether the end point of the center line of the i-th lane is connected to the starting point of the center line of the j-th lane. If so, the value is 1, otherwise it is 0.

[0035] In this embodiment, the camera internal and external parameter matrix is: 1) The camera internal and external parameter matrix is ​​used for the conversion between the camera coordinate system and the world coordinate system, and between the camera coordinate system and the image coordinate system.

[0036] 2) In most cases, the camera's intrinsic and extrinsic parameter matrices remain unchanged over the entire time period.

[0037] 3) Each camera has only one internal and external parameter matrix, which is used to convert the image observed by the camera into information in the world coordinate system.

[0038] like Figure 6 As shown, in this embodiment, the image feature map generation submodule 11 is described below: Using the pre-trained ResNet50 and FPN structures and parameters, each image is processed to extract the 0th, 1st, 2nd, and 3rd layer feature maps of ResNet50. The sizes of the four feature maps of ResNet50 are: 56×56×256, 28×28×512, 14×14×1024, and 7×7×2048. Due to the different sizes and number of channels of the feature maps, the number of channels needs to be unified through FPN. The FPN network unifies the sizes of the above four feature maps to: 56×56×256, 28×28×256, 14×14×256, and 7×7×256. For the three feature maps of size 28×28, 14×14, and 7×7, the bilinear interpolation method is used to upsample to 56×56 and finally add them in the channel dimension to obtain the feature map of the image, with a final size of 56×56×256.

[0039] like Figure 6 As shown, in this embodiment, the purpose of the image feature map generation submodule 11 is to convert and splice the feature maps of multiple cameras at the same time according to the rules to generate a BEV feature map based on the vehicle coordinate system. The GKT method is used to project multiple surrounding feature maps to a unified BEV view based on the vehicle coordinate system to form a first BEV feature map. The dimensions of the first BEV feature map are 500×300×256, where 500 and 300 are determined according to the predetermined resolution and BEV perception range. 256 is determined according to the number of channels of the feature map after extraction and fusion by ResNet50 and FPN. At the same time, the surrounding image of the vehicle at a certain moment is converted into the first BEV feature map based on the vehicle coordinate system. The first BEV feature map is a representation of the vehicle environment, has the advantages of avoiding occlusion, etc., and is suitable for the fusion of multi-source data. The first BEV feature map has spatial characteristics, and its coverage range is consistent with the BEV perception range. For the relative position in the two-dimensional space within the BEV perception range, the corresponding feature vector can be found according to the resolution and the range of the BEV feature map.

[0040] like Figure 6 As shown, in this embodiment, the purpose of using navigation map data is to increase the representation ability of the BEV feature map of the map information of the area in the form of prior information. The navigation map data is a collection of a series of vector (points, polylines and polygons) data, and its position information is the world coordinate. Therefore, the SD MAP data should first be converted to the local coordinate system according to the vehicle posture information and the BEV feature space range. In addition, it is also necessary to encode the vector information into the BEV space, that is, rasterize it, and this processing is implemented by semantic map and One-hot encoding. Finally, the first BEV feature map and the second BEV feature map are stacked in the direction of the number of channels, and then projected into the target number of channels through the perceptron to form a fused feature map.

[0041] The input of BEV spatial feature extraction module 1 is the surround camera image data with timestamp, and the output is the fused feature map.

[0042] like Figure 7 As shown, in this embodiment, the feature information interaction module 2 is composed of a feature information interaction submodule 21 based on a transformer structure with deformable attention and a feature information propagation submodule 22 based on a graph convolutional network (GCN) structure. It should be noted that the feature information interaction module 2 can be reused multiple times to fully realize the fusion and transmission of information.

[0043] like Figure 7 As shown, the feature information interaction submodule 21 is described below: The deformable cross self-attention mechanism is adopted. The query matrix is ​​the embedding of the lane centerline with a dimension of 100 × 256, which represents the features of each lane centerline. Each element in the query matrix is ​​a learnable and updateable numerical value. The key-value pair matrix is ​​the feature map generated by the BEV spatial feature extraction module 1 with a dimension of 600 × 300 × 256. The feature information interaction submodule 21 uses a multi-head attention mechanism with 8 heads. The dimension of each attention head is 32, which is calculated based on the dimension of the query matrix and the number of heads. Deformable attention calculation is performed for each attention head. First, the query matrix is ​​linearly projected to convert its dimension to 1×32. Then, the feature map is projected through the linear projection matrix to convert its dimension to 600×300 ×32. For each query, the position coordinates of the reference point and the coordinates and weights of the corresponding four sampling points are determined by linear projection. For each attention head, the features of the reference point and the sampling point are interacted with the fused feature map to obtain a feature vector. The feature vectors obtained from all attention heads are superimposed and converted to a 1×256 feature vector by linear projection. By stacking the feature information interaction submodule 21 multiple times, the fusion and transmission of information can be fully realized.

[0044] like Figure 7 As shown, the characteristic information propagation submodule 22 is described below: Based on the graph convolutional network (GCN) structure, the input is the output of the feature information interaction submodule 21, that is, the query matrix containing the BEV features. The lane centerlines and their topological relationships are represented by a graph. The adjacency matrix represents the connection relationship between the lane centerlines, and its size is 100×100. The elements in the adjacency matrix indicate whether the two lane centerlines are connected, 1 indicates connected, and 0 indicates disconnected. The degree matrix is ​​a diagonal matrix, and the elements on the diagonal represent the number of nodes connected to the point. The model (i.e., graph convolutional network) uses learnable linear projection matrices WO and W1, which are calculated based on the adjacency matrix and the degree matrix. The topological relationship fusion between different lane centerlines is calculated through the GCN network, and the final output feature vector dimension is 1 × 256.

[0045] Through the processing of the feature information interaction module 2, the fusion of the lane centerline geometry information and the topological relationship information is realized. At the same time, since the output of the feature information interaction module 2 is still a 100 × 256 query matrix, the feature information interaction module 2 can be stacked multiple times in sequence, and the parameters of the linear transformation matrix are the same each time. Considering the calculation time and cost, it is temporarily set to 5 times in this embodiment.

[0046] In this embodiment, through the processing of the feature information interaction module 2, the query matrix has fully integrated the BEV features and lane centerline topology information at the current moment. The main purpose of the centerline and topology relationship prediction module 3 is to decode this information through a multi-layer perceptron (MLP) to achieve the prediction of lane centerline coordinate points and the prediction of the topological relationship between lane centerlines. The following is a detailed description of the centerline and topology relationship prediction module 3: 1) Prediction of lane centerline coordinates: like Figure 8 As shown, each lane centerline consists of 10 ordered nodes, and each node is represented by a set of three-dimensional coordinates. The first multilayer perceptron 31 (i.e., MLP1) is used to calculate the coordinates of the 10 ordered nodes of each lane centerline. The input of the first multilayer perceptron 31 is the query matrix output by the feature information interaction module 2, which is 100×256 in size. The first multilayer perceptron 31 is a two-dimensional matrix of 256×30 in size, in which each element is a pre-trained parameter. Through the calculation of the first multilayer perceptron 31, the coordinates of each lane centerline node are predicted.

[0047] 2) Lane centerline confidence prediction: like Figure 8As shown, the second multilayer perceptron 32 (i.e., MLP2) is used to calculate the confidence of the center line of each lane, and its input is still the query matrix output by the feature information interaction module 2, with a size of 100×256. The second multilayer perceptron 32 is a two-dimensional matrix with a size of 256×1, in which each element is a pre-trained parameter. The confidence of the center line of each lane is predicted through the calculation of the second multilayer perceptron 32.

[0048] 3) Lane centerline topological relationship prediction: like Figure 8 As shown, this task is processed using three multi-layer perceptrons, namely the third multi-layer perceptron 33 (i.e., MLP3), the fourth multi-layer perceptron 34 (i.e., MLP4), and the fifth multi-layer perceptron 35 (i.e., MLP5). The third multi-layer perceptron 33 is used to project the lane centerline features into a low-dimensional space, and the input query matrix size of the third multi-layer perceptron 33 is 100×256. The third multi-layer perceptron 33 is a two-dimensional matrix with a size of 256×128, in which each element is a pre-trained parameter. The features of any two lane centerlines after dimensionality reduction in the query matrix are stacked in the channel direction to form 256-channel features. The fourth multi-layer perceptron 34 is used to predict the topological relationship between the two lane centerlines. The fourth multi-layer perceptron 34 is a two-dimensional matrix with a size of 256×1, in which each element is a pre-trained parameter. The fifth multi-layer perceptron 35 determines whether the two lane centerlines are connected by setting a threshold according to the output value of the fourth multi-layer perceptron 34, that is, according to the stacking result, predicts the topological relationship between the two centerlines and outputs the lane centerline topological relationship.

[0049] In summary, this method can realize the function of generating lane centerlines and centerline topological connection relationships based on surround view image data and navigation map data, and the result is the necessary information for downstream assisted driving planning and control.

[0050] This method automatically formulates rules through training data to achieve end-to-end lane centerline generation and its topological relationship from the perspective of BEV. This method can improve generalization and robustness, and can handle complex scenarios such as where the number of lanes changes and transition sections from straight sections to curved sections.

[0051] In this embodiment, an electronic device includes a memory and a controller, wherein the memory stores a computer-readable program, and when the computer-readable program is called by the controller, it can execute the steps of the lane centerline and topological relationship generation method as described in this embodiment.

[0052] In this embodiment, a vehicle adopts the lane centerline and topological relationship generation system as described in this embodiment.

[0053] In this embodiment, a storage medium stores a computer-readable program, and when the computer-readable program is called, it can execute the steps of the lane centerline and topological relationship generation method as described in this embodiment.

[0054] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A lane centerline and topological relationship generation system, characterized in that: include: A feature extraction module (1) is used to convert the vehicle surround video data into a first BEV feature map in a vehicle body coordinate system through a feature extraction network, convert the navigation map data into a local coordinate system according to the vehicle posture information and the BEV feature space range, and perform BEV encoding processing to obtain a second BEV feature map, and perform feature map fusion on the first BEV feature map and the second BEV feature map to obtain a fused feature map; The feature information interaction module (2) uses a deformable cross self-attention mechanism to process the feature map at each moment, and outputs a query matrix that integrates the feature map information, wherein the query matrix is ​​a plurality of feature vectors that represent the lane centerline information; and calculates the topological relationship fusion between different lane centerlines through the GCN network based on the query matrix that integrates the feature map information, and outputs a query matrix that integrates the lane centerline topological relationship, and uses the query matrix that integrates the lane centerline topological relationship as the query matrix for the next feature information interaction, thereby achieving full information interaction through multiple rounds of iterations; The centerline and topological relationship prediction module (3) predicts the three-dimensional coordinates of the nodes constituting the centerline of each lane and the confidence level of the centerline of the lane based on a query matrix that incorporates the topological relationship of the centerlines of the lanes; and predicts the topological relationship of two centerlines based on the query matrix that incorporates the topological relationship of the centerlines of the lanes.

2. The lane centerline and topological relationship generation system according to claim 1, characterized in that: The BEV spatial feature extraction module (1) comprises: The image feature map generation submodule (11) uses a convolutional neural network to extract features from the image captured by each camera at each moment, and performs multi-scale fusion to generate a feature map; through an inverse perspective method and in combination with the internal and external parameters of the camera, the feature maps of multiple cameras at the same moment are converted into a BEV space, and are fused to output the first BEV feature map in the ego-vehicle coordinate system; A navigation map encoding submodule (12) converts the navigation map data into a local coordinate system according to the vehicle position information and the BEV feature space range, and performs BEV encoding processing to obtain a second BEV feature map; The data fusion submodule (13) stacks the first BEV feature map and the second BEV feature map in the direction of the number of channels, and then projects them into the target number of channels through a perceptron to form a fused feature map.

3. The lane centerline and topological relationship generation system according to claim 1, characterized in that: The feature information interaction module (2) comprises: The feature information interaction submodule (21) uses a deformable cross self-attention mechanism to process the feature map at each moment, and outputs a query matrix that integrates the feature map information, wherein the query matrix is ​​a plurality of feature vectors representing the lane centerline information; The feature information propagation submodule (22) receives the query matrix output by the feature information interaction submodule (21) that integrates the feature map information, calculates the topological relationship fusion between the center lines of different lanes through the GCN network, outputs the query matrix that integrates the topological relationship of the center lines of the lanes, and uses the query matrix that integrates the topological relationship of the center lines of the lanes as the query matrix for the next feature information interaction, thereby achieving full information interaction through multiple rounds of iterations.

4. The lane centerline and topological relationship generation system according to claim 1, characterized in that: The centerline and topological relationship prediction module (3) comprises: A first multi-layer perceptron (31) predicts the coordinates of each lane centerline node based on a query matrix output by the feature information interaction module (2) that incorporates the lane centerline topological relationship; A second multi-layer perceptron (32) predicts the confidence of each lane centerline based on the query matrix output by the feature information interaction module (2) that incorporates the topological relationship of the lane centerlines; A third multi-layer perceptron (33) is used to reduce the dimension of the query matrix output by the feature information interaction module (2) and integrating the topological relationship of the lane center lines; A fourth multi-layer perceptron (34) is used to stack the features of any two lane center lines in the query matrix after dimension reduction in the channel direction; The fifth multi-layer perceptron (35) predicts the topological relationship between the center lines of the two lanes based on the stacking results.

5. A method for generating lane centerline and topological relationship, characterized in that: The following steps are involved: S1: Acquire vehicle surround video data, and convert the vehicle surround video data into a first BEV feature map in a vehicle body coordinate system through a feature extraction network; acquire navigation map data, convert the navigation map data into a local coordinate system according to the vehicle posture information and the BEV feature space range, and perform BEV encoding processing to obtain a second BEV feature map, and perform feature map fusion on the first BEV feature map and the second BEV feature map to obtain a fused feature map; S2: Use a deformable cross self-attention mechanism to process the feature map at each moment, and output a query matrix that integrates the feature map information, where the query matrix is ​​a plurality of feature vectors that represent the lane centerline information; and calculate the topological relationship fusion between different lane centerlines through the GCN network based on the query matrix that integrates the feature map information, output a query matrix that integrates the lane centerline topological relationship, and use the query matrix that integrates the lane centerline topological relationship as the query matrix for the next feature information interaction, and achieve full information interaction through multiple rounds of iterations; S3: Predict the three-dimensional coordinates of the constituent nodes of each lane centerline and the confidence level of the lane centerline based on a query matrix that incorporates the topological relationship of the lane centerlines; and predict the topological relationship of two centerlines based on a query matrix that incorporates the topological relationship of the lane centerlines.

6. The method for generating lane center lines and topological relationships according to claim 5, characterized in that: The S1 is specifically: For the images taken by each camera at each moment, a convolutional neural network is used to extract features and perform multi-scale fusion to generate a feature map. The feature maps of multiple cameras at the same moment are converted to the BEV space through the inverse perspective method and combined with the internal and external parameters of the camera, and then fused to output the first BEV feature map in the vehicle coordinate system. The navigation map data is converted into a local coordinate system according to the self-vehicle posture information and the BEV feature space range, and BEV encoding is performed to obtain a second BEV feature map; The first BEV feature map and the second BEV feature map are stacked in the direction of the number of channels, and then projected into the target number of channels through the perceptron to form a fused feature map.

7. The method for generating lane center lines and topological relationships according to claim 5, characterized in that: The S3 includes: The coordinates of each lane centerline node are predicted based on the query matrix that incorporates the lane centerline topological relationship; Predict the confidence of each lane centerline based on the query matrix that incorporates the topological relationship of the lane centerlines; Reduce the dimension of the query matrix that incorporates the lane centerline topology; Stack the reduced-dimensional features of any two lane centerlines in the query matrix in the channel direction; Based on the stacking results, predict the topological relationship between the center lines of the two lanes.

8. An electronic device, characterized in that: It includes a memory and a controller, wherein the memory stores a computer-readable program, and when the computer-readable program is called by the controller, it can execute the steps of the lane centerline and topological relationship generation method as described in any one of claims 5 to 7.

9. A medium, characterized in that: A computer-readable program is stored therein, and when the computer-readable program is called by the controller, the steps of the lane centerline and topological relationship generation method as described in any one of claims 5 to 7 can be executed.

Images