A transformer-based high-definition map real-time prediction method and system

Abstract

The application discloses a high-precision map real-time prediction method based on a Transformer, and comprises the following steps: acquiring multiple image data containing road categories from multiple perspective cameras, and camera internal and external parameters corresponding to each image data and calibration parameters of a self vehicle; acquiring a projection matrix corresponding to each image data by using matrix multiplication, and pre-processing all the image data to obtain multiple pre-processed image data; inputting the pre-processed image data and the projection matrix corresponding to each image data into a high-precision map real-time prediction model pre-trained to obtain a prediction result corresponding to each road category; processing a segmentation graph corresponding to each road category by using a point non-maximum suppression method to obtain a key point set corresponding to all the road categories; and decoding the key point set corresponding to each road category and an offset graph and a distance graph corresponding to the road category to obtain a decoded detection result.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of deep learning and visual perception, and more particularly, relates to a high-precision map real-time prediction method and system based on a Transformer. BACKGROUND

[0002] High-precision map real-time prediction plays a crucial role in the field of autonomous driving research. Accurate perception of the road is often the first step in downstream tasks such as route planning, vehicle steering, and lane keeping. Therefore, high-precision map real-time prediction research is an important part of current deep learning, and has great research potential and application value. In addition, high-precision maps usually contain various road category information such as lane lines, sidewalks, and road edges.

[0003] Currently, high-precision map real-time prediction algorithms are mainly divided into three methods. The first method is to not project 2D image features, but to directly detect road structure information related to high-precision maps in the Bird's Eye View (BEV) space, and then use the camera's internal and external parameters and pose information to directly convert the detection results to the Bird's Eye View space. The second method is to use Inverse Perspective Mapping (IPM) to map 2D image feature information to the Bird's Eye View space according to the camera's internal and external parameters and pose information, to obtain a Bird's Eye View feature map, and then detect road structure information of the high-precision map through the Bird's Eye View feature map. The third method is a depth estimation-based method, which adds pixel-level depth information to elevate the position information of each pixel to three dimensions, and then projects it to the camera coordinate system and to the Bird's Eye View space using the camera's internal parameters, to obtain a Bird's Eye View feature map, and finally detects road structure information of the high-precision map through the Bird's Eye View feature map.

[0004] However, the above-mentioned several existing high-precision map prediction methods all have some technical problems that cannot be ignored. First, directly converting the detection results in the image coordinate system to the Bird's Eye View space does not make good use of the high-level semantic information of the image features, making the model lack robustness, and simply splicing the detection results corresponding to multiple cameras makes the image features corresponding to multiple cameras not efficiently fused. Second, as a simplified camera model operation, Inverse Perspective Mapping introduces position errors in the conversion process, and in the face of complex scenes and inaccurate camera parameters, the detection may not be good. Third, since the current depth estimation model still has the technical problem of inaccurate prediction, a small depth estimation deviation may introduce a large spatial conversion position error, which will further affect the accuracy of high-precision map detection. SUMMARY

[0005] In view of the above defects or improvement needs of the prior art, the present application provides a high-precision map real-time prediction method and system based on a Transformer, which aims to solve the technical problems that the existing method directly converts the road detection result in the image space to the bird's-eye view space, which easily leads to a lack of robustness of the model, insufficient interaction of feature information between multiple camera perspectives, and ultimately reduces the prediction accuracy of the high-precision map, and the technical problems that the method based on inverse perspective change simplifies the camera model and cannot handle complex driving scenes, and the method based on depth estimation is too dependent on the accuracy of depth information, which easily leads to inaccurate depth estimation and reduces the accuracy of converting image features to the bird's-eye view space.

[0006] To achieve the above-mentioned purpose, according to one aspect of the present application, a high-precision map real-time prediction method based on a Transformer is provided, comprising the following steps:

[0007] (1) Obtain multiple image data containing road categories, and the camera internal and external parameters corresponding to each image data and the calibration parameters of the ego vehicle from multiple perspective cameras, obtain the projection matrix corresponding to each image data according to the pre-established bird's-eye view space coordinate system, the camera internal and external parameters corresponding to each image data and the calibration parameters of the ego vehicle, and utilize matrix multiplication, and pre-process all image data to obtain multiple pre-processed image data.

[0008] (2) Input all pre-processed image data in step (1) and the projection matrix corresponding to each image data into a pre-trained high-precision map real-time prediction model to obtain the prediction result corresponding to each road category, which includes a segmentation map, an offset map and a distance map.

[0009] (3) Process the segmentation map corresponding to each road category obtained in step (2) using a point non-maximum suppression method to obtain a key point set corresponding to all road categories, decode the key point set corresponding to each road category and the offset map and distance map corresponding to the road category obtained in step (2) to obtain a decoded detection result, and splice the detection results corresponding to all road categories to obtain the final high-precision map prediction result.

[0010] Preferably, the camera internal and external parameters include the camera internal parameters required for the projection of the camera coordinate system to the image coordinate system and the external parameters required for the projection of the camera coordinate system to the ego vehicle coordinate system, and the ego vehicle calibration data mainly include but are not limited to the position and orientation of the ego vehicle.

[0011] The road categories include three types, i.e., lane lines, sidewalks and road edges.

[0012] Preferably, in step (1), the process of calculating the projection matrix corresponding to each image data according to the pre-established bird's-eye view spatial coordinate system, the camera intrinsic and extrinsic parameters corresponding to each image data and the vehicle calibration parameters, and using matrix multiplication, specifically involves first multiplying the camera intrinsic and extrinsic parameters to obtain the projection matrix that transforms the image coordinate system to the vehicle coordinate system; then, according to the pre-established bird's-eye view spatial coordinate system, converting the vehicle calibration parameters into the projection matrix that transforms the camera coordinate system to the bird's-eye view spatial coordinate system; and finally, multiplying the two projection matrices to obtain the projection matrix that transforms the image coordinate system to the bird's-eye view spatial coordinate system.

[0013] In step (1), preprocessing operations are performed on the multiple input image data, including scaling and normalization operations. The scaling operation uses bilinear interpolation to scale the image data from its original size to 448×800×3, and the normalization operation removes the average brightness value from the image data.

[0014] Preferably, the high-precision map real-time detection model comprises three parts connected in sequence: a feature extraction backbone network, a bird's-eye view ground construction network, and a prediction head network;

[0015] The feature extraction backbone network consists of a standard ResNet network and an axial feature pyramid module, with the following specific structure:

[0016] The ResNet network takes multiple image data from all viewpoints as input, performs standard feature extraction operations on it, and outputs a dimensional [database name missing]. as well as The size is a feature tensor, where W represents the width of each image data, which takes a value of 800, H represents the height of each image data, which takes a value of 448, and the number of channels for each image data is 3;

[0017] Axial feature pyramid module, whose input is the output of the ResNet network. as well as The feature tensors of size are arranged in descending order to obtain layer feature tensors. Axial self-attention is calculated for each layer feature tensor, and then axial cross-attention is calculated between each layer and the updated feature tensor from the previous layer. The final output is... as well as The feature tensor of size;

[0018] The bird's-eye view ground construction network consists of N decoding layers connected in series, where N ranges from 1 to 3. The input to each decoding layer is a query feature of size 100×50×256 and the output of the feature pyramid module. The feature tensor of size 100×50×256 is first processed by axial self-attention for all query features to obtain an updated feature tensor of size 100×50×256. Then, for each query feature in the updated feature tensor of size 100×50×256, it is classified as either visible or invisible based on whether the query feature is visible when projected onto the camera viewpoint. Finally, the visible query features and the feature pyramid module output are processed together. as well as Multi-view sampling attention computation is performed on the feature tensor of size N1×256 to obtain the updated visible query features. Then, the invisible query features and the output of the feature pyramid module are processed. Global cross-attention is performed on the feature tensor of size N2×256 to obtain the updated invisible query features, where N1+N2=5000. Finally, all query features are input into the fully connected layer network to obtain a feature tensor of size 100×50×256, which is used as the input of the next decoding layer. Thus, the final output of the bird's-eye view ground construction network is a feature tensor of size 100×50×256.

[0019] The prediction head network consists of three parallel parts: lane line prediction head, pedestrian prediction head, and curb prediction head. All prediction heads have the same structure. Each prediction head is composed of three identical parallel network branches, and each network branch is composed of M convolutional layers connected sequentially. A feature tensor of size 100×50×256 is input into the three prediction heads and upsampling is performed to obtain three prediction results. Each prediction result includes a segmentation map of size 400×200×2, an offset map of size 400×200×4, and a distance map of size 400×200×2, where the value of M ranges from 2 to 4.

[0020] Preferably, the high-precision map real-time prediction model is trained through the following steps:

[0021] (2-1) Obtain the high-precision map autonomous driving dataset, the camera intrinsic and extrinsic parameters corresponding to each image data, and the vehicle calibration parameters. Preprocess the high-precision map autonomous driving dataset to obtain the preprocessed high-precision map autonomous driving dataset, and divide it into training set and validation set according to the proportion. Based on the pre-established bird's-eye view spatial coordinate system, the camera intrinsic and extrinsic parameters corresponding to each image data, and the vehicle calibration parameters, obtain the projection matrix corresponding to each image data using matrix multiplication.

[0022] (2-2) Input the training set obtained in step (2-1) into the pre-trained ResNet model to obtain the image features at the l-th scale corresponding to each viewpoint v. Axial attention is calculated for the multi-scale image features corresponding to each viewpoint v to obtain the multi-scale image features corresponding to that viewpoint. Then, cross-scale fusion is performed on the multi-scale image features corresponding to this viewpoint to obtain updated multi-scale image features. Where v∈{1,...,V}, V represents the number of camera viewpoints, and l∈{1,...,L}, L represents the number of scales of image features;

[0023] (2-3) Initialize a set of rasterized and learnable query features Q, with size H. B ×W B ×C, where H B W represents the length of the bird's-eye view plane. B C represents the width of the bird's-eye view plane, and Q represents each query feature. i The number of channels, where i represents the query feature index, and i∈{1,...,H} B W B}, i.e., Q i The size is 1×C, representing the 3D coordinates corresponding to each query feature. The projection matrix corresponding to each image obtained in step (2-1) is calculated to obtain the 2D coordinates of each query feature after projection in the image coordinate system. Based on the relationship between the 2D projection coordinates of all query features and the size of the image data, all query features are divided into visible query features and invisible query features. Multi-view sampling attention calculation (MSA) is performed on each visible query feature and its corresponding 2D coordinates and the multi-view multi-scale image features obtained in step (2-2) to obtain the updated visible query features.

[0024] (2-4) Perform position embedding layer calculation on the multi-scale image features corresponding to the multiple camera viewpoints updated in step (2-2) to obtain 3D image features. Perform hole query completion (DQC) calculation on each invisible query feature and 3D image features to obtain updated invisible query features. Concatenate the visible query features obtained in step (2-3) and the updated invisible query features according to the predefined corresponding 3D position coordinates to obtain the bird's-eye view feature map.

[0025] (2-5) Input the bird's-eye view feature map obtained in step (2-4) into the prediction head network to obtain the prediction results corresponding to each road category. Each prediction result includes a segmentation map, an offset map, and a distance map.

[0026] (2-6) Calculate the loss function according to the segmentation prediction results, the offset prediction results and the distance prediction results of all road categories obtained in step (2-5), and iteratively train the high-definition map real-time detection model using the loss function until the high-definition map real-time detection model converges, thereby obtaining the trained high-definition map real-time detection model.

[0027] Preferably, the axial attention operation CA is performed on the image features at each scale in step (2-2), and each query feature only performs attention calculation with the keys in the same row or the same column;

[0028] The cross-scale feature fusion operation in step (2-2) is also an axial attention calculation, and the image features at the lth scale corresponding to the vth view When the updated image features at the l+1th scale corresponding to the vth view are taken as query features, When the updated image features at the l+1th scale corresponding to the vth view are taken as query features, are up-sampled by bilinear interpolation, and the size and are the same, and the specific operation is shown in the following formula:

[0029]

[0030]

[0031] Where CA represents the axial attention operation, and UP represents the up-sampling operation.

[0032] Preferably, in step (2-3), each query feature Q i represents the feature information of the grid cell located at p i = (x i , y i ) in the aerial view space coordinate plane. And in the entire coordinate system conversion information processing process, the corresponding position reference point of each Q i is fixed, so for each Q i , the corresponding 3D position coordinates are (x i *s, y i *s, h), where s is the resolution of the aerial view space, and h is the ground height in the aerial view space;

[0033] In step (2-3), the corresponding 3D coordinates of each query feature Q i are calculated and the projection matrix corresponding to each image obtained in step (2-1) to obtain the 2D coordinates of each query feature in the image coordinate system after projection. The process is as follows: the projection matrix corresponding to the vth view camera obtained in step (2-1) and Calculations are performed to obtain the projection point of the image plane corresponding to the v-th camera. The specific operation is shown in the following formula.

[0034]

[0035] Where p v This represents the projection matrix required to transform the v-th camera from 3D space to the image plane;

[0036] In step (2-3), the process of dividing all query features into visible and invisible query features specifically involves, based on all projection points I... i,v Whether it is visible in the corresponding viewpoint, Q i Query features are categorized as either visible or invisible. Visible query features are visible within at least one camera view, while invisible query features are invisible across all camera views. Used to represent Q i The visibility query feature is Q, which camera viewpoints are visible. x ={Q i ||U i |>0}, the invisible query characteristic is Q y ={Q i ||U i |=0},Q=Q x +Q y .

[0037] In step (2-3), the process of performing multi-view sampling attention calculation on the visible query features and the multi-view, multi-scale image features obtained in step (2-2) is specifically as follows:

[0038]

[0039] Among them, Q i ∈Q x , Representing I i,v Normalized coordinates, F v Let represent the multi-scale image features of the v-th viewpoint, l represent the index value of the image feature scale, and k represent the index value of the sampling point. L represents the number of feature scales, and K represents the number of sampling points. and These represent the k-th sampling point at... Attention weights and sampling point offsets on the feature map, where W represents the learnable network parameters. This represents normalized coordinates. Mapped to On the feature map.

[0040] Preferably, the process of calculating the corresponding location embedding layer for the multi-view, multi-scale image features obtained in step (2-2) in step (2-4) to obtain 3D image features is as follows: First, calculate the Lth layer size of all viewpoints as H. L ×W L ×C image features By piecing together an ideal cylinder, and establishing a 3D coordinate system with the cylinder's center point O as the origin, the 3D position coordinates of all image pixels p can be obtained. Then, let Let be the projection points of p from the frontal and top-down viewpoints, respectively. α is the projection point of p. B The angle between point O and the projection point of point O from the top viewpoint, where β is p R The angle between point O and its projection point in the normal viewing angle. Approximated by calculating sinα, cosα, and tanβ. and and Then p O = (sinα,cosα,tanβ), the specific formula is shown below.

[0041]

[0042] Then, the 3D position coordinates p corresponding to all the obtained pixels are... O Feeding into a fully connected layer to obtain a ring-shaped position embedding layer. Dimensions are H L ×W L ×C, where C is the number of channels, which is consistent with the number of image feature channels;

[0043] Finally, the resulting ring-shaped position embedding layer and image features Add them together to obtain a view-sensitive 3D feature map. The specific formula is shown below.

[0044]

[0045] The process of performing hole completion calculations on invisible query features and view-sensitive 3D feature maps in steps (2-4) is specifically shown in the following formula:

[0046]

[0047] Q i ∈Q y k represents the index value of the pixel in the image data, W, A v and A k These are all learnable network parameters, and DQC stands for Hollow Query Completion.

[0048] Preferably, in step (2-5), the aerial view feature map obtained in step (2-4) is respectively fed into three prediction head networks to obtain three groups of prediction results, each of which contains a segmentation map S with a size of 400x200x2, an offset map T with a size of 400x200x4, and a distance map D with a size of 400x200x2, wherein the segmentation map is used to record the area of the lane line foreground point, the offset map is used to record the offset vector between each foreground point and the adjacent node, and the distance map saves the distance scalar of each foreground point and the endpoint of the lane line where it is located.

[0049] The total loss function Loss is:

[0050] Loss = Loss ped + Loss lane + Loss boundary

[0051] The lane line loss function Loss lane is:

[0052] Loss lane = Loss seg + Loss reg

[0053]

[0054]

[0055] wherein, is a segmentation map supervision signal generated by lane line ground truth labeling, with a size of 400x200x2, is a distance map supervision signal generated by lane line ground truth labeling, with a size of 400x200x2, is a distance map supervision signal generated by lane line ground truth labeling, with a size of 400x200x4. The segmentation map uses an OHEM loss function, and the positive and negative sample ratio is 15:1, while the offset map and the distance map use a SMOOTH-L1 loss function.

[0056] The sidewalk loss function Loss ped and the curb loss function Loss boundary are the same as the lane line loss function Loss lane .

[0057] According to another aspect of the present application, a high-precision map real-time prediction system based on a Transformer is provided, comprising:

[0058] The first module is configured to obtain multiple image data containing road categories from multiple perspective cameras, and camera internal and external parameters corresponding to each image data and calibration parameters of the ego vehicle, obtain a projection matrix corresponding to each image data according to a pre-established bird's eye view space coordinate system, camera internal and external parameters corresponding to each image data and calibration parameters of the ego vehicle, and utilize matrix multiplication, and pre-process all image data to obtain multiple pre-processed image data.

[0059] The second module is configured to input all image data pre-processed by the first module and the projection matrix corresponding to each image data into a pre-trained high-definition map real-time prediction model to obtain a prediction result corresponding to each road category, which includes a segmentation map, an offset map and a distance map.

[0060] The third module is configured to process the segmentation map corresponding to each road category obtained by the second module using a point non-maximum suppression method to obtain a key point set corresponding to all road categories, decode the key point set corresponding to each road category and the offset map and the distance map corresponding to the road category obtained by the second module to obtain a decoded detection result, and splice the detection results corresponding to all road categories to obtain a final high-definition map prediction result.

[0061] Overall, the above technical solutions conceived by the present application can achieve the following beneficial effects compared with the prior art:

[0062] (1) The present application adopts step (2-3) in the training process of the high-definition map real-time prediction model, which realizes a new bird's eye view projection method through pre-defined grid query features, obtains more accurate bird's eye view features, and therefore solves the technical problems of the prior art that directly convert road detection results in the image space to the bird's eye view space, resulting in a lack of robustness of the model, and the inability to interact information between multiple perspective cameras.

[0063] (2) The present application adopts step (2-3) in the training process of the high-definition map real-time prediction model, which realizes the inverse projection of the bird's eye view space to the image space through the query feature, better utilizes the pre-defined three-dimensional space information, and thus obtains the bird's eye view space feature information, solving the technical problem of the prior art that simplifies the camera model based on the inverse perspective transformation, assuming that the target projection plane is smooth and flat, resulting in the inability to handle complex driving scenarios.

[0064] (3) The application solves the technical problem that the existing depth estimation-based method is affected by depth information, resulting in inaccurate depth estimation and reducing the accuracy of image feature conversion to bird's eye view space, by using step (2-3) in the high-precision map real-time prediction model training process, which dynamically predicts the position offset of the sampling point using the network model through multi-view sampling attention operation, thereby adapting to the change in depth information caused by the change in ground height in the bird's eye view space.

[0065] (4) The application solves the technical problem that the existing depth estimation-based method is affected by depth information, resulting in inaccurate depth estimation and reducing the accuracy of image feature conversion to bird's eye view space, by using step (2-4) in the high-precision map real-time prediction model training process, which divides the query features into visible query features and invisible query features, and then performs hole query completion calculation on the invisible query features, thereby enhancing the generalization ability of the model for the invisible area centered on the ego vehicle, and improving the accuracy of road detection.

[0066] (5) The application solves the technical problem of feature position misalignment caused by up-sampling in the traditional addition operation by using step (2-2) in the high-precision map real-time prediction model training process, which performs axial attention calculation on the multi-scale image features corresponding to multiple camera perspectives, and uses axial feature attention calculation instead of the traditional feature addition operation when performing cross-scale feature fusion, thereby enabling the obtained image features to have global information, which is consistent with the topological structure characteristics of the road, and avoiding the technical problem of feature position misalignment caused by up-sampling in the traditional addition operation. BRIEF DESCRIPTION OF DRAWINGS

[0067] Figure 1 is a flowchart of the high-precision map real-time prediction method based on the Transformer of the application;

[0068] Figure 2 is a structural schematic diagram of the high-precision map real-time detection model of the application.

[0069] Figure 3 is a structural schematic diagram of the axial feature pyramid in the high-precision map real-time detection model of the application.

[0070] Figure 4 is a structural schematic diagram of the hole query completion in the high-precision map real-time detection model of the application. DETAILED DESCRIPTION

[0071] In order to make the purpose, technical scheme and advantages of the application more clear, the application will be further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the application and do not limit the application. In addition, the technical features involved in each embodiment of the application described below can be combined with each other as long as they do not conflict with each other.

[0072] In order to solve the technical problem of inaccurate projection of Bird's eye view (BEV) features in the field of high-definition map real-time prediction, the application provides a high-definition map real-time prediction method based on a Transformer. Through the proposed multi-view sampling attention mechanism, the pre-defined grid-based learnable query features and image features are efficiently interacted to generate a BEV feature map with more accurate spatial position information, and the generalization ability of the model for the invisible area centered on the ego vehicle in the BEV space is further improved, thereby improving the learning ability and detection performance of the network for multiple road class information in the BEV space, and combining the detected multiple road class information into the final required high-definition map.

[0073] As shown in Figure 1 The application provides a high-definition map real-time prediction method based on a Transformer, which comprises the following steps:

[0074] (1) Obtain multiple image data containing road classes from multiple perspective cameras, and the camera internal and external parameters corresponding to each image data and the calibration parameters of the ego vehicle. According to the pre-established BEV space coordinate system, the camera internal and external parameters corresponding to each image data and the calibration parameters of the ego vehicle, and by using matrix multiplication, obtain the projection matrix corresponding to each image data (which is used to realize the conversion between the image coordinate system and the BEV space coordinate system), and pre-process all image data to obtain multiple pre-processed image data.

[0075] Specifically, the camera internal and external parameters include the camera internal parameters required for the projection of the camera coordinate system to the image coordinate system and the camera external parameters required for the projection of the camera coordinate system to the ego vehicle coordinate system, and the ego vehicle calibration data mainly include but are not limited to the position and orientation of the ego vehicle. The road classes include three types, i.e. lane lines, sidewalks and road edges.

[0076] In this step, according to the pre-established BEV space coordinate system, the camera internal and external parameters corresponding to each image data and the calibration parameters of the ego vehicle, and by using matrix multiplication, the process of calculating the projection matrix corresponding to each image data is as follows: first, multiply the camera internal and external parameters to obtain the projection matrix for the conversion of the image coordinate system to the ego vehicle coordinate system, then convert the calibration parameters of the ego vehicle into the projection matrix for the conversion of the camera coordinate system to the BEV space coordinate system according to the pre-established BEV space coordinate system, and finally multiply the two obtained projection matrices to obtain the projection matrix for the conversion of the image coordinate system to the BEV space coordinate system.

[0077] In this step, the input multiple image data is preprocessed, including scaling and normalization operation, the scaling operation is to scale the image data from the original size to 448*800*3 by using the bilinear interpolation method, which can reduce the calculation amount of the network model, and the purpose of the normalization operation is to remove the average brightness value in the image data, after removing the common part of all samples, the individual difference between samples can be highlighted.

[0078] (2) input all image data preprocessed in step (1) and the projection matrix corresponding to each image data into the high-precision map real-time prediction model pre-trained to obtain the prediction result corresponding to each road category, the prediction result including a segmentation map, an offset map and a distance map.

[0079] As shown in Figure 2 , the high-precision map real-time detection model of the application comprises a feature extraction backbone network, an aerial view ground construction network and a prediction head network connected in sequence.

[0080] The feature extraction backbone network comprises a standard ResNet network and an axial feature pyramid module, and the specific structure is as follows:

[0081] The ResNet network takes multiple image data (which are RGB images) of all viewing angles as input, performs standard feature extraction operation on the image data, and outputs a feature tensor with a dimension of and a size of , wherein W represents the width of each image data, which is 800, H represents the height of each image data, which is 448, and the channel number of each image data is 3.

[0082] The axial feature pyramid module, as shown in Figure 3 , takes the feature tensor with a size of and output by the ResNet network as input, arranges them in order from large to small according to the size, and obtains a layer feature tensor. Each layer feature tensor is subjected to axial self-attention calculation, and then subjected to axial cross-attention calculation with the updated feature tensor of the previous layer, and finally outputs a feature tensor with a size of and .

[0083] The aerial view ground construction network is composed of N (wherein the value range of N is 1 to 3, and the preferred value is 2) decoding layers connected in series. The input of each decoding layer is a query (Query) feature with a size of 100*50*256 and a feature tensor with a size of The size of the feature tensor is first calculated by performing axial self-attention calculation on all query features to obtain updated query features with a size of 100*50*256, then each query feature of the updated 100*50*256 size feature tensor is traversed, and according to the result of whether the query feature projection is visible in the camera view, the query feature is divided into a visible query feature or a dilated query feature, then the visible query feature and the output of the feature pyramid module are input to perform multi-view sampling attention calculation on the 100*50*256 size feature tensor to obtain updated visible query features with a size of N1*256, then the dilated query feature and the output of the feature pyramid module are input to perform global cross-attention calculation on the 100*50*256 size feature tensor to obtain updated dilated query features with a size of N2*256, wherein N1+N2=5000, finally, all query features are input into a fully connected layer network to obtain a 100*50*256 size feature tensor as the input of the next decoding layer, and thus the bird's eye view ground construction network finally outputs a 100*50*256 size feature tensor. and and and

[0084] The prediction head network includes three parts of a lane line prediction head, a sidewalk prediction head and a road edge prediction head in parallel, the structures of all prediction heads are completely same, each prediction head is composed of three same parallel network branches, each network branch is sequentially connected by M (wherein the value range of M is 2 to 4, and preferably 3) convolution layers, the 100*50*256 size feature tensor is input into the three prediction heads, and up-sampling operation is performed to obtain three prediction results, each prediction result includes a 400*200*2 size segmentation map, a 400*200*4 size offset map and a 400*200*2 size distance map.

[0085] Specifically, the high-definition map real-time prediction model of the application is trained by the following steps:

[0086] (2-1) Obtain a high-definition map automatic driving data set, camera internal and external parameters corresponding to each image data and calibration parameters of the ego vehicle, pre-process the high-definition map automatic driving data set to obtain a pre-processed high-definition map automatic driving data set, and divide it into a training set and a validation set according to a predetermined proportion, according to the pre-established bird's eye view space coordinate system, the camera internal and external parameters corresponding to each image data and the calibration parameters of the ego vehicle, and using matrix multiplication to obtain the projection matrix corresponding to each image data.

[0087] Specifically, the high-precision map automatic driving dataset used in this step is the nuScenes dataset, which is collected in four areas of Boston Harbor, Singapore's Queenstown, North and Holland Village, a total of 1000 automatic driving scenes, including rainy days, nights and fog, etc. When training and testing, the official dataset is used for division, and the ratio of 4.7:1 is used to divide the training set and the test set, that is, the training set has a total of 28130 timestamp samples, and the validation set has a total of 6019 timestamp samples. Each timestamp includes 6 camera image data, camera internal and external parameters, and a predefined bird's eye view space coordinate system to obtain the projection matrix corresponding to each image data. In addition, this step only focuses on three types of road information: lane lines, sidewalks, and road edges.

[0088] Note that the image preprocessing method, projection matrix calculation method used in this step, and the above step (1) are exactly the same, so they will not be repeated here.

[0089] (2-2) Input the training set obtained in step (2-1) into the pre-trained ResNet model to obtain the image features of each view v corresponding to the lth scale The axis attention calculation is performed on the multi-scale image features corresponding to each view v to obtain the multi-scale image features corresponding to the view And the multi-scale image features corresponding to the view are cross-scale fused to obtain the updated multi-scale image features Where v∈{1,...,V}, V represents the number of camera views, l∈{1,...,L}, L represents the number of image feature scales;

[0090] Specifically, this step performs axis attention operation (Columnar Attention, abbreviated as CA) on the image features at each scale, that is, self-attention calculation is performed, and each query feature (Query) only performs attention calculation with the same row or column key (Key).

[0091] The cross-scale feature fusion operation in this step also performs the same axis attention calculation, the difference is that this operation performs cross-attention calculation, and the image features of the lth scale corresponding to the vth view When the updated image features of the l+1th scale corresponding to the vth view When the query feature, the key and the value (Value) are used to perform bilinear interpolation on Upsampling (abbreviated as UP) operation, and the size and The same, so as to avoid the technical problem of position misalignment introduced by the traditional upsampling operation, the specific operation is shown in the following formula:

[0092]

[0093]

[0094] where CA denotes an axial attention operation, and UP denotes an up-sampling operation.

[0095] (2-3) Initialize a set of rasterized and learnable query features Q with size H B x W B x C, where H B represents the length of the aerial view plane, W B represents the width of the aerial view plane, and C represents the number of channels of each query feature Q i , i represents the index of the query feature, and i ∈ {1,..., H B W B}, i.e., the size of Q i is 1 x C, and the 3D coordinates corresponding to each query feature and the projection matrix corresponding to each image obtained in step (2-1) to obtain the 2D coordinates of each query feature in the image coordinate system after projection, and according to the relationship between the obtained 2D projection coordinates corresponding to all query features and the size of the image data, all query features are divided into visible query features and invisible query features (where the visible query features are visible at least in one camera view, and the invisible query features are invisible in all camera views), and for each visible query feature and the corresponding 2D coordinates and the multi-view multi-scale image features obtained in step (2-2), multi-view sampling attention calculation (Multi-view Sampling Attention, referred to as MSA) is performed to obtain updated visible query features.

[0096] Specifically, each query feature Q i represents the feature information of the grid cell located at p i = (x i , y i ) in the aerial view space coordinate plane. And in the entire coordinate system conversion information processing process, the position reference point corresponding to each Q i is fixed and unchanged, so for each Q i , the corresponding 3D position coordinates are (x i * s, y i * s, h), where s is the resolution of the aerial view space, and h is the ground height in the aerial view space.

[0097] In this step (2-3), each query feature Q icorresponding 3D coordinates and the projection matrix corresponding to each image obtained in step (2-1) to obtain the 2D coordinates of each query feature after projection in the image coordinate system. The process is specifically as follows: the projection matrix corresponding to the vth view camera obtained in step (2-1) and the projection matrix corresponding to the image plane of the vth camera are calculated to obtain the projection point of the image plane of the vth camera The specific operation is shown in the following formula,

[0098]

[0099] where p v represents the projection matrix required for the vth camera to convert from 3D space to the image plane.

[0100] In this step (2-3), all query features are divided into visible query features and invisible query features. The process is specifically as follows: according to all projection points I i,v , it is determined whether the corresponding view is visible, and Q i is classified into visible query features or invisible query features. The visible query features are visible in at least one camera view, and the invisible query features are invisible in all camera views. Let be used to represent Q i which camera views are visible, then the visible query features are Q x ={Q i ||U i |>0}, and the invisible query features are Q y ={Q i ||U i |=0}, Q=Q x +Q y .

[0101] In this step (2-3), the visible query features and the multi-view multi-scale image features obtained in step (2-2) are subjected to multi-view sampling attention calculation. The process is specifically as follows:

[0102]

[0103] where Q i ∈Q x , represents the normalized coordinates of I i,v , F v represents the multi-scale image features of the vth view, l represents the index value of the image feature scale, and k represents the index value of the sampling point. L represents the number of feature scales, and K represents the number of sampling points. and respectively represent the kth sampling point in ​The attention weight on the feature map and the sampling point offset, W represents the learnable network parameters, representing the normalized coordinates mapped to the feature map, MSA represents multi-view sampling attention calculation.

[0104] The advantage of the present sub-step is that the projection process of the query feature is re-examined, and all query features are divided into two categories according to the projection results. This is because the present sub-step finds that due to the fixed installation mode of multiple cameras, a small area centered on the ego vehicle is invisible in all images, and the query features located in this area cannot be projected onto the image through the camera internal and external parameters. Note that this area does not exist in most cases. Therefore, it has little effect on the 3D target detection task. However, the feature learning ability of this area has a great influence on the lane line detection task, because there is a large amount of lane line structured information around the ego vehicle, which is crucial for the final detection performance.

[0105] (2-4) Perform position embedding layer calculation on the multi-scale image features corresponding to the multiple camera perspectives updated in step (2-2) to obtain 3D image features (which are 3D image features sensitive to perspective), perform dilate query completion (DQC for short) calculation on each invisible query feature and the 3D image features to obtain updated invisible query features, and splice the visible query features obtained in step (2-3) and the updated invisible query features according to the pre-defined corresponding 3D position coordinates to obtain an aerial view feature map, as shown in the specific process of Figure 4 .

[0106] In the present step (2-4), the multi-perspective multi-scale image features obtained in step (2-2) are calculated by the corresponding position embedding layer to obtain 3D image features. The process is as follows: first, all image features of size H L ×W L ×C of the Lth layer of all perspectives are spliced into an ideal cylinder, and a 3D space coordinate system is established with the center point O of the cylinder as the origin, to obtain the 3D position coordinates of all image pixel points p Then, let be the projection points of p in the front view and the top view, respectively. α is the included angle between p B and the projection point of point O in the top view, and β is the included angle between p R and the projection point of point O in the front view. By calculating sinα, cosα, tanβ, the following approximations are obtained and and Then p O = (sinα, cosα, tanβ), the specific formula is as follows,

[0107]

[0108] Subsequently, the 3D position coordinates p O of all the pixel points obtained are sent into a full connection layer to obtain a ring position embedding layer with a size of H L × W l × C, wherein C is the number of channels, and the number of image feature channels is consistent. Finally, the ring position embedding layer obtained is added to the image features to obtain a view angle sensitive 3D feature map The specific formula is as follows,

[0109]

[0110] The process of performing the hole query completion calculation on the invisible query features and the view angle sensitive 3D feature map in this step (2-4) is specifically shown in the following formula:

[0111]

[0112] wherein Q i ∈ Q y , k represents an index value of the image data pixel points, W, A v and A k are all learnable network parameters, and DQC represents a hole query completion operation.

[0113] (2-5) The bird's eye view feature map obtained in step (2-4) is input into a prediction head network (i.e., a lane line prediction head, a sidewalk prediction head and a road edge prediction head) to obtain prediction results corresponding to each road category, respectively. Each prediction result includes a segmentation map, an offset map and a distance map.

[0114] Specifically, for the three road categories of lane lines, sidewalks and road edges, three completely same prediction heads are designed in this step, which respectively focus on different categories of road information. Referring to the RCLane algorithm framework, the bird's eye view feature map obtained in step (2-4) is sent into the three prediction head networks to obtain three groups of prediction results, each of which includes a segmentation map S with a size of 400×200×2, an offset map T with a size of 400×200×4, and a distance map D with a size of 400×200×2. The segmentation map is used to record the area of the lane line foreground point, the offset map is used to record the offset vector between each foreground point and the adjacent node, and the distance map saves the distance scalar of each foreground point and the end point of the lane line.

[0115] (2-6) Calculate the loss function according to the segmentation prediction result, the offset prediction result and the distance prediction result of all road categories obtained in step (2-5), and iteratively train the high-definition map real-time detection model using the loss function until the high-definition map real-time detection model converges, thereby obtaining the trained high-definition map real-time detection model.

[0116] wherein the lane line loss function Loss lane is:

[0117] Loss lane = Loss seg + Loss reg

[0118]

[0119]

[0120] wherein, is a segmentation map supervision signal generated by lane line ground truth labeling, with a size of 400x200x2, is a distance map supervision signal generated by lane line ground truth labeling, with a size of 400x200x2, is a distance map supervision signal generated by lane line ground truth labeling, with a size of 400x200x4. The segmentation map uses an OHEM loss function, with a positive to negative sample ratio of 15:1, while the offset map and the distance map use a SMOOTH-L1 loss function. The sidewalk loss function Loss ped and the curb loss function Loss boundary and the lane line loss function Loss lane are the same, therefore, the total loss function Loss is:

[0121] Loss = Loss ped + Loss lane + Loss boundary

[0122] (3) Use a point non-maximum suppression (Point NMS) method to process the segmentation map corresponding to each road category obtained in step (2) to obtain a key point set corresponding to each road category, and decode the key point set corresponding to each road category, and the offset map and the distance map corresponding to the road category obtained in step (2) to obtain a decoded detection result. Splice the detection results corresponding to all road categories (i.e. lane lines, sidewalks and curbs) to obtain the final high-definition map prediction result.

[0123] Specifically, the decoding algorithm used in this step is the decoding algorithm proposed by Xu et al. in the paper RCLane: Relay Chain Prediction for Lane Detection published in 2022, which is specifically from Xu S, Cai X, Zhao B, et al. RCLane: Relay Chain Prediction for Lane Detection [C] / / Computer Vision-ECCV 2022: 17th European Conference, Proceedings, Part XXXVIII. Cham: Springer Nature Switzerland, 2022: 461-477.

[0124] Test results

[0125] In order to illustrate the effectiveness of the method of the present application and the improvement of the detection accuracy of high-precision maps, comparative experiments were carried out with several mainstream network models on the nuScenes autonomous driving data set. The evaluation indicators involved in the test process of the present method include: (1) mean Intersection over Union: this value evaluates the coincidence degree of each category of road structure in the final high-precision map generation result in the semantic information layer and the true value. The more the predicted area and the true value area of all road categories coincide, the larger the value is, and the better the method effect is; (2) mean average precision (mAP for short): this value measures the instance detection of the model in all categories. It evaluates the coincidence degree between the predicted road matching the target true value road and the true value road. The more the predicted road instance information and the true value coincide, the larger the value is, and the better the method effect is.

[0126] The following Table 1 and Table 2 show the comparison of the present application and other methods on the nuScenes data set:

[0127] Table 1

[0128]

[0129] Table 2

[0130]

[0131]

[0132] As can be seen from Table 1 and Table 2, after the new aerial view feature generation mode of converting the aerial view space to the image coordinate system, the mIoU and mAP values of the method reach 42.0% and 27.8% respectively. This shows that compared with the methods of other models, the method starts from the perspective of aerial view feature projection, fully utilizes the three-dimensional position prior knowledge of the aerial view space through the pre-defined manner of query features, realizes a more accurate projection method, dynamically focuses on the overlapping areas between multiple perspective cameras through multi-perspective sampling attention, efficiently performs information fusion between image features corresponding to multiple cameras, performs global cross-attention calculation between invisible query features and image features, improves the generalization ability of the model to the invisible area centered on the ego vehicle in the driving scene, and further improves the accuracy of real-time prediction of the high-definition map.

[0133] Those skilled in the art can easily understand that the above description is only preferred embodiments of the present application, and is not used to limit the present application, and any modifications, equivalent replacements and improvements made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A real-time prediction method for high-precision maps based on Transformer, characterized in that, Includes the following steps: (1) Acquire multiple image data containing road categories from multiple perspective cameras, as well as the camera intrinsic and extrinsic parameters and vehicle calibration parameters corresponding to each image data. Based on the pre-established bird's-eye view spatial coordinate system, the camera intrinsic and extrinsic parameters and vehicle calibration parameters corresponding to each image data, obtain the projection matrix corresponding to each image data using matrix multiplication, and preprocess all image data to obtain multiple preprocessed image data. (2) Input all the image data after preprocessing in step (1) and the projection matrix corresponding to each image data into the pre-trained high-precision map real-time prediction model to obtain the prediction results corresponding to each road category. The prediction results include segmentation map, offset map and distance map. The high-precision map real-time detection model consists of three parts connected in sequence: feature extraction backbone network, bird's-eye view ground construction network and prediction head network. The feature extraction backbone network consists of a standard ResNet network and an axial feature pyramid module, with the following specific structure: The ResNet network takes multiple image data from all viewpoints as input, performs standard feature extraction operations on it, and outputs a dimensional [database name missing]. as well as The size is a feature tensor, where W represents the width of each image data, which takes a value of 800, H represents the height of each image data, which takes a value of 448, and the number of channels for each image data is 3; Axial feature pyramid module, whose input is the output of the ResNet network. as well as The feature tensors of size are arranged in descending order of size to obtain the layer feature tensors; Each layer of feature tensors undergoes axial self-attention calculation, and then axial cross-attention calculation is performed between each layer and the updated feature tensor from the previous layer. The final output is... as well as The feature tensor of size; The bird's-eye view ground construction network consists of N decoding layers connected in series, where N ranges from 1 to 3. The input to each decoding layer is a query feature of size 100×50×256 and the output of the feature pyramid module. The feature tensor of size 100×50×256 is first processed by axial self-attention for all query features to obtain an updated feature tensor of size 100×50×256. Then, for each query feature in the updated feature tensor of size 100×50×256, it is classified as either visible or invisible based on whether the query feature is visible when projected onto the camera viewpoint. Finally, the visible query features and the feature pyramid module output are processed together. as well as Multi-view sampling attention computation is performed on the feature tensor of size N1×256 to obtain the updated visible query features. Then, the invisible query features and the output of the feature pyramid module are processed. Global cross-attention is performed on the feature tensor of size N2×256 to obtain the updated invisible query features, where N1+N2=5000. Finally, all query features are input into the fully connected layer network to obtain a feature tensor of size 100×50×256, which is used as the input of the next decoding layer. Thus, the final output of the bird's-eye view ground construction network is a feature tensor of size 100×50×256. The prediction head network consists of three parallel parts: lane line prediction head, sidewalk prediction head, and curb prediction head. All prediction heads have the same structure. Each prediction head consists of three identical parallel network branches. Each network branch is composed of M convolutional layers connected sequentially. A feature tensor of size 100×50×256 is input into the three prediction heads and upsampling is performed to obtain three prediction results. Each prediction result includes a segmentation map of size 400×200×2, an offset map of size 400×200×4, and a distance map of size 400×200×2, where the value of M ranges from 2 to 4. (3) The point non-maximum suppression method is used to process the segmentation map corresponding to each road category obtained in step (2) to obtain the key point set corresponding to all road categories. The key point set corresponding to each road category, as well as the offset map and distance map corresponding to the road category obtained in step (2), are decoded to obtain the decoded detection result. The detection results corresponding to all road categories are stitched together to obtain the final high-precision map prediction result.

2. The real-time prediction method for high-precision maps based on Transformer according to claim 1, characterized in that, The camera intrinsic and extrinsic parameters include the camera intrinsic parameters required to project the camera coordinate system onto the image coordinate system and the extrinsic parameters required to project the camera coordinate system onto the vehicle coordinate system. The vehicle calibration data mainly includes, but is not limited to, the vehicle's position and orientation. There are three types of roads: lane lines, sidewalks, and curbs.

3. The real-time prediction method for high-precision maps based on Transformer according to claim 1 or 2, characterized in that, In step (1), the process of calculating the projection matrix corresponding to each image data according to the pre-established bird's-eye view spatial coordinate system, the camera intrinsic and extrinsic parameters corresponding to each image data and the vehicle calibration parameters, and using matrix multiplication is as follows: First, the camera intrinsic and extrinsic parameters are multiplied to obtain the projection matrix of the image coordinate system to the vehicle coordinate system. Then, according to the pre-established bird's-eye view spatial coordinate system, the vehicle calibration parameters are converted into the projection matrix of the camera coordinate system to the bird's-eye view spatial coordinate system. Finally, the two projection matrices are multiplied to obtain the projection matrix of the image coordinate system to the bird's-eye view spatial coordinate system. In step (1), preprocessing operations are performed on the multiple input image data, including scaling and normalization operations. The scaling operation uses bilinear interpolation to scale the image data from its original size to 448×800×3, and the normalization operation removes the average brightness value from the image data.

4. The real-time prediction method for high-precision maps based on Transformer according to claim 3, characterized in that, The high-precision map real-time prediction model is trained through the following steps: (2-1) Obtain the high-precision map autonomous driving dataset, the camera intrinsic and extrinsic parameters corresponding to each image data, and the vehicle calibration parameters. Preprocess the high-precision map autonomous driving dataset to obtain the preprocessed high-precision map autonomous driving dataset, and divide it into training set and validation set according to the proportion. Based on the pre-established bird's-eye view spatial coordinate system, the camera intrinsic and extrinsic parameters corresponding to each image data, and the vehicle calibration parameters, obtain the projection matrix corresponding to each image data using matrix multiplication. (2-2) Input the training set obtained in step (2-1) into the pre-trained ResNet model to obtain the image features at the l-th scale corresponding to each viewpoint v. Axial attention is calculated for the multi-scale image features corresponding to each viewpoint v to obtain the multi-scale image features corresponding to that viewpoint. Then, cross-scale fusion is performed on the multi-scale image features corresponding to this viewpoint to obtain updated multi-scale image features. Where v∈{1,...,V}, V represents the number of camera viewpoints, and l∈{1,...,L}, L represents the number of scales of image features; (2-3) Initialize a set of rasterized and learnable query features Q, with size H. B ×W B ×C, where H B W represents the length of the bird's-eye view plane. B C represents the width of the bird's-eye view plane, and Q represents each query feature. i The number of channels, where i represents the index value of the query feature, and i∈{1,...,H} B W B }, i.e., Q i The size is 1×C, representing the 3D coordinates corresponding to each query feature. The projection matrix corresponding to each image obtained in step (2-1) is calculated to obtain the 2D coordinates of each query feature after projection in the image coordinate system. Based on the relationship between the 2D projection coordinates of all query features and the size of the image data, all query features are divided into visible query features and invisible query features. Multi-view sampling attention calculation (MSA) is performed on each visible query feature and its corresponding 2D coordinates and the multi-view multi-scale image features obtained in step (2-2) to obtain the updated visible query features. (2-4) Perform position embedding layer calculation on the multi-scale image features corresponding to multiple camera viewpoints after the update in step (2-2) to obtain 3D image features. Perform hole query completion (DQC) calculation on each invisible query feature and 3D image features to obtain the updated invisible query features. Concatenate the visible query features obtained in step (2-3) and the updated invisible query features according to the predefined corresponding 3D position coordinates to obtain the bird's-eye view feature map. (2-5) Input the bird's-eye view feature map obtained in step (2-4) into the prediction head network to obtain the prediction results corresponding to each road category. Each prediction result includes a segmentation map, an offset map, and a distance map. (2-6) Calculate the loss function based on the segmentation prediction results, offset prediction results and distance prediction results of all road categories obtained in step (2-5), and use the loss function to iteratively train the high-precision map real-time detection model until the high-precision map real-time detection model converges, thereby obtaining the trained high-precision map real-time detection model.

5. The real-time prediction method for high-precision maps based on Transformer according to claim 4, characterized in that, In step (2-2), axial attention operation CA is performed on the image features at each scale, and attention is calculated only between each query feature and the keys in the same row or column. The cross-scale feature fusion operation in step (2-2) also involves axial attention calculation, which combines the image features of the l-th scale corresponding to the v-th viewpoint. Treat it as a query feature, and update the image features at the (l+1)th scale corresponding to the vth viewpoint. Treating them as keys and values, using bilinear interpolation pairs Perform upsampling operation, and the size and The specific operation is as follows: Where CA represents axial attention operation and UP represents upsampling operation.

6. The real-time prediction method for high-precision maps based on Transformer according to claim 5, characterized in that, In steps (2-3), each query feature Q i This indicates that the center of the bird's-eye view is located in the spatial coordinate plane of the image. i =(x i ,y i The characteristic information of the grid cells; and in the information processing of the entire coordinate system transformation, each Q i The corresponding reference points are all fixed, therefore for each Q... i Its corresponding 3D position coordinates For (x) i *s,y i *s,h), where s is the resolution of the bird's-eye view space and h is the ground height in the bird's-eye view space; In steps (2-3), each query feature Q will be... i Corresponding 3D coordinates The process of calculating the projection matrix corresponding to each image obtained in step (2-1) to obtain the 2D coordinates of each query feature in the image coordinate system after projection is specifically as follows: the projection matrix corresponding to the v-th viewpoint camera obtained in step (2-1) and... Calculations are performed to obtain the projection point of the image plane corresponding to the v-th camera. The specific operation is shown in the following formula. Where p v This represents the projection matrix required to transform the v-th camera from 3D space to the image plane; In step (2-3), the process of dividing all query features into visible and invisible query features specifically involves, based on all projection points I... i,v Whether it is visible in the corresponding viewpoint, Q i Query features are categorized into visible or invisible features. Visible features are visible within at least one camera view, while invisible features are invisible across all camera views. Used to represent Q i The visibility query feature is Q, which camera viewpoints are visible. x ={Q i ||U i |>0}, the invisible query characteristic is Q y ={Q i ||U i |=0},Q=Q x +Q y ; In step (2-3), the process of performing multi-view sampling attention calculation on the visible query features and the multi-view, multi-scale image features obtained in step (2-2) is specifically as follows: Among them, Q i ∈Q x , Representing I i,v Normalized coordinates, F v Let l represent the multi-scale image features of the v-th viewpoint, l represent the index value of the image feature scale, k represent the index value of the sampling point, L represent the number of feature scales, and K represent the number of sampling points. and These represent the k-th sampling point at... Attention weights and sampling point offsets on the feature map, where W represents the learnable network parameters. This represents normalized coordinates. Mapped to On the feature map.

7. The real-time prediction method for high-precision maps based on Transformer according to claim 6, characterized in that, In step (2-4), the multi-view, multi-scale image features obtained in step (2-2) are processed to calculate the corresponding location embedding layers to obtain 3D image features. Specifically, this process is as follows: First, the Lth layer of all viewpoints is of size H. L ×W L ×C image features By piecing together an ideal cylinder, and establishing a 3D coordinate system with the cylinder's center point O as the origin, the 3D position coordinates of all image pixels p can be obtained. Then, let Let be the projection points of p from the frontal and top-down viewpoints, respectively; α is the projection point of p. B The angle between point O and the projection point of point O from the top viewpoint, where β is p R The angle between point O and its projection point in the frontal view is approximated by calculating sinα, cosα, and tanβ. and and Then p O = (sinα,cosα,tanβ), the specific formula is shown below. Then, the 3D position coordinates p corresponding to all the obtained pixels are... O Feeding into a fully connected layer to obtain a ring-shaped position embedding layer. Dimensions are H L ×W L ×C, where C is the number of channels, which is consistent with the number of image feature channels; Finally, the resulting ring-shaped position embedding layer and image features Add them together to obtain a viewpoint-sensitive 3D feature map. The specific formula is shown below. The process of performing hole completion calculations on invisible query features and view-sensitive 3D feature maps in steps (2-4) is specifically shown in the following formula: Q i ∈Q y k represents the index value of the pixel in the image data, W, A v and A k These are all learnable network parameters, and DQC stands for Hollow Query Completion.

8. The real-time prediction method for high-precision maps based on Transformer according to claim 7, characterized in that, In steps (2-5), the bird's-eye view feature maps obtained in steps (2-4) are fed into three prediction head networks to obtain three sets of prediction results. Each set of prediction results includes a segmentation map S with a size of 400×200×2, an offset map T with a size of 400×200×4, and a distance map D with a size of 400×200×2. The segmentation map is used to record the region of the foreground point of the lane line, the offset map is used to record the offset vector between each foreground point and its adjacent nodes, and the distance map stores the distance scalar between each foreground point and the endpoint of the lane line. The total loss function is: Loss=Loss ped +Loss lane +Loss boundary The lane line loss function Loss lane for: Loss lane =Loss seg +Loss reg in, The segmentation map supervision signal is generated from the lane line ground truth annotation, with dimensions of 400×200×2. The distance map monitoring signal is generated from the lane line ground truth annotations, with dimensions of 400×200×2. The distance map supervision signal is generated from the ground truth annotation of lane lines, with a size of 400×200×4; the segmentation map uses the OHEM loss function with a positive to negative sample ratio of 15:1, while the offset map and distance map use the SMOOTH-L1 loss function; Sidewalk Loss Function ped and the loss function along the road boundary Lane line loss function Loss lane same.

9. A real-time prediction system for high-precision maps based on Transformer, used to implement the real-time prediction method for high-precision maps based on Transformer as described in claim 1, characterized in that, The high-precision map real-time prediction system includes: The first module is used to acquire multiple image data containing road categories from multiple perspective cameras, as well as the camera intrinsic and extrinsic parameters and vehicle calibration parameters corresponding to each image data. Based on the pre-established bird's-eye view spatial coordinate system, the camera intrinsic and extrinsic parameters and vehicle calibration parameters corresponding to each image data, the module uses matrix multiplication to obtain the projection matrix corresponding to each image data, and preprocesses all image data to obtain multiple preprocessed image data. The second module is used to input all the image data preprocessed by the first module, as well as the projection matrix corresponding to each image data, into the pre-trained high-precision map real-time prediction model to obtain the prediction results corresponding to each road category. The prediction results include segmentation map, offset map and distance map. The third module is used to process the segmentation map corresponding to each road category obtained by the second module using the point non-maximum suppression method to obtain the key point set corresponding to all road categories. The key point set corresponding to each road category, as well as the offset map and distance map corresponding to that road category obtained by the second module, are decoded to obtain the decoded detection result. The detection results corresponding to all road categories are stitched together to obtain the final high-precision map prediction result.

Images