BEV adaptive online mapping method, system and device and storage medium

Abstract

The invention discloses a BEV self-adaptive online mapping method, system and device and a storage medium, which are corresponding schemes, and in the scheme, from the perspective of joint perception of looking-around pinholes and fisheye images, image features are extracted by using multi-view pinholes and fisheye images, pinhole and fisheye camera distortion models are considered, BEV features are generated through self-adaptive mapping, and the BEV features are extracted by using the multi-view pinholes and fisheye images. And self-adaptive lane attention is designed for a BEV online mapping task, so that the joint perception difficulty caused by fisheye image distortion is solved, and the prediction precision and stability of online mapping are improved. Generally speaking, the accuracy and reliability of the automatic driving system based on the low-cost fisheye camera scheme for sensing the topological structure of the lane under the complex real street scene can be remarkably improved, and a solid technical guarantee is provided for downstream path planning and control decision making.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving perception technology, and in particular to a BEV adaptive online mapping method, system, device and storage medium. Background Technology

[0002] With the rapid development of technologies such as artificial intelligence, deep learning, and large-scale models, autonomous driving technology has become a major research hotspot. Traditional vehicles are being redefined by artificial intelligence, becoming intelligent terminals. While improving people's travel safety, autonomous driving technology also greatly improves traffic efficiency and alleviates traffic congestion. Bird's Eye View (BEV) perception technology is a key component of the autonomous driving technology stack. Among them, BEV online mapping technology is an important part of the perception module, aiming to construct the lane topology structure from the BEV's perspective in real time through multi-sensor data input, for use by the downstream planning layer of the autonomous driving system.

[0003] However, most current online mapping methods for autonomous driving BEVs (Battery Electric Vehicles) employ surround-view pinhole camera solutions, with few using fisheye camera solutions. Compared to pinhole cameras, fisheye cameras offer a larger field of view (FOV), typically requiring only four cameras to cover the environment surrounding the autonomous vehicle. They are also more cost-effective and are commonly used for near-field perception in low-speed scenarios such as parking. However, the severe radial distortion of fisheye cameras leads to an extremely non-uniform distribution of pixel information density in their image space. This distribution exhibits a "central clustering effect": the central region of the image corresponds to a narrow frontal field of view, where pixel information is dense, distortion is minimal, and object shapes are well preserved, providing high-confidence details for perception. Conversely, closer to the image edges correspond to extremely wide lateral and rear-lateral field of view, where vast spaces in the physical world are compressed into a limited number of pixels, resulting in a drastic dilution of information density in edge pixels. Simultaneously, these pixels also bear the most severe nonlinear distortion, causing severe distortion of object geometry and contextual relationships, thus forming a highly complex and difficult-to-decouple distorted context within a limited number of pixels. This characteristic of "information density decay" and "distortion complexity surge" coexisting from the center to the edge has a strong impact on BEV visual perception solutions based on fisheye cameras, especially for technical solutions that require fine prediction, such as online mapping of BEVs. Therefore, it is extremely challenging to optimize and adapt the algorithm design for distorted fisheye images.

[0004] Existing BEV online mapping and related methods for joint perception of pinhole and fisheye images still have many limitations and are difficult to meet the practical application requirements of joint perception of pinhole and fisheye distorted images, as well as high precision, high efficiency, low cost and strong adaptability. For example: (1) Chinese invention patent application CN119887508A, "A Bird's-eye View Fusion Generation Method Based on Bilinear Interpolation", uses camera intrinsic parameters to transform the pixel coordinate system to the image coordinate system and the image coordinate system to the ground coordinate system. By using the intrinsic and extrinsic parameters of the fisheye camera, the ground coordinates are transformed into the pixel coordinates of the fisheye camera, and the distance weight is calculated. The coordinate distance weight is combined to obtain a new bilinear interpolation weight and obtain the pixels of the BEV image. The SSIM similarity comparison is performed using pixel blocks with rich line segment features in the fusion area of ​​the fisheye image, and the mapping table weight is adjusted. The calculation of the entire mapping table in this method depends entirely on the absolute accuracy of the camera's intrinsic parameters (distortion coefficients, focal length, optical center) and extrinsic parameters (rotation, translation matrix). Any tiny calibration error (such as parameter drift caused by vibration during vehicle operation) will lead to incorrect mapping relationships, resulting in ghosting, misalignment, or breakage of BEV images. The system has no self-correction capability. (2) In the Chinese invention patent application CN119314147A, "Method and System for Obstacle and Occupancy Grid Prediction Using a Fisheye Camera", multiple frames of images are first received from multiple fisheye cameras. Image features and depth probability distributions are extracted from these multiple frames of images through the Backbone, and BEV viewpoint transformation is performed on these image features to generate BEV features. Subsequently, obstacle prediction and occupancy grid prediction tasks are completed based on the generated BEV features. For the processing of the surround-view fisheye images, the Lift-Splat-Shoot (LSS) method is adopted, and fisheye distortion coefficients are explicitly added for distortion removal. Inference is accelerated by offline pre-calculation lookup table. However, LSS's projection process relies on the camera's intrinsic and extrinsic parameters, which constitute a static, geometric transformation. It assumes the entire world is rigid and all objects are stationary. In real-world scenarios, there are numerous moving objects. LSS cannot distinguish between static and moving objects; it projects feature points on moving objects onto incorrect geographical locations based on camera parameters, resulting in inaccurate BEV features and impacting the performance of downstream perception tasks.(3) Chinese invention patent application CN120558252A proposes a method for constructing a local vector map for autonomous driving based on a high-precision map and fusion perception. This method includes high-precision map information preprocessing and the construction of a fusion perception network. In the high-precision map information preprocessing, the original high-precision map data is processed into a vectorized representation suitable for model input. The fusion perception network includes a feature extraction backbone network, a BEV perspective transformation network, a multimodal BEV feature fusion network, and a map element decoding module. In the map element decoding module, the processed high-precision map elements interact with the BEV bird's-eye view features from the sensor. In the local map model training, query denoising is introduced as an additional training task, and query ranking selection is introduced to accelerate model convergence. However, this method relies on the accuracy of the high-precision map. Due to frequent changes in road infrastructure, the update speed of the SD Map (measured monthly or weekly) is much slower than the actual speed of road changes. When encountering temporary detours, newly constructed intersections, or road construction, outdated SD Map priors can mislead the system's perception results and even cause danger. On the other hand, this method uses seven pinhole camera images as input, but does not configure a fisheye camera with a larger FOV or adapt it for fisheye cameras, resulting in relatively poor compatibility.

[0005] In view of this, the present invention is hereby proposed. Summary of the Invention

[0006] The purpose of this invention is to provide a BEV adaptive online mapping method, system, device, and storage medium that can meet the practical application requirements of joint perception of pinhole images and fisheye distortion images, as well as high precision, high efficiency, low cost, and strong adaptability.

[0007] The objective of this invention is achieved through the following technical solution: A BEV adaptive online mapping method includes: Step 1: Extract features from the input pinhole camera and surround-view fisheye camera images to obtain corresponding feature maps; perturb the BEV reference points using an adaptive offset strategy, and determine the corresponding two-dimensional reference points for each image based on the perturbated BEV reference points; weight the corresponding feature maps based on the two-dimensional reference points to obtain weighted feature maps; combine the attention mechanism to aggregate the weighted feature maps corresponding to all images into BEV features. Step 2: Model the semantic associations of the learnable map instance query representations corresponding to each lane line segment through a self-attention mechanism to obtain the updated map instance query representations for each lane line segment; for each lane line segment, calculate the sampling weights of each reference point based on the distance relationship between its corresponding reference points, and combine the updated map instance query representations with BEV features to obtain the corresponding feature responses through an attention mechanism; use the feature vectors of each lane line segment for classification and regression to obtain the lane category and point cloud in the BEV space for each lane line segment; Step 3: Combine the connection relationships between lane line segments, the lane category of each lane line segment, and the point cloud in the BEV space to construct a complete BEV lane topology map.

[0008] A BEV adaptive online mapping system for implementing the aforementioned method includes: The encoding module is used to extract features from the input images from the pinhole camera and the surround-view fisheye camera to obtain the corresponding feature maps; the BEV reference points are perturbed by an adaptive offset strategy, and the two-dimensional reference points corresponding to each image are determined based on the perturbed BEV reference points; the corresponding feature maps are weighted based on the two-dimensional reference points to obtain weighted feature maps; and the weighted feature maps corresponding to all images are aggregated into BEV features by combining an attention mechanism. The decoding module is used to model the semantic associations of the learnable map instance query representations corresponding to each lane line segment through a self-attention mechanism, thereby obtaining the updated map instance query representations for each lane line segment. For each lane line segment, the sampling weights of each reference point are calculated based on the distance relationship between its corresponding reference points, and the corresponding feature responses are obtained through an attention mechanism by combining the updated map instance query representations with BEV features. The feature vectors of each lane line segment are used for classification and regression to obtain the lane category and point cloud in the BEV space for each lane line segment. The mapping module is used to construct a complete BEV lane topology map by combining the connection relationships between lane line segments, the lane category of each lane line segment, and the point cloud in the BEV space.

[0009] A processing device includes: one or more processors; and a memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

[0010] A readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned method.

[0011] As can be seen from the technical solution provided by the present invention, from the perspective of joint perception of surround-view pinhole and fisheye images, image features are extracted using multi-view pinhole and fisheye images. Considering the distortion models of pinhole and fisheye cameras, adaptive mapping is used to generate BEV features. Furthermore, adaptive lane attention is designed for the online mapping task of BEVs, solving the difficulties in joint perception caused by fisheye image distortion and improving the prediction accuracy and stability of online mapping. Overall, the present invention can significantly improve the accuracy and reliability of lane topology perception in autonomous driving systems based on low-cost fisheye camera solutions under complex real-world street scenes, providing a solid technical guarantee for downstream path planning and control decisions. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a flowchart of a BEV adaptive online mapping method provided in an embodiment of the present invention.

[0014] Figure 2 This is a schematic diagram of the overall architecture of a BEV adaptive online mapping method provided in an embodiment of the present invention.

[0015] Figure 3 This is a schematic diagram of an adaptive BEV module provided in an embodiment of the present invention.

[0016] Figure 4 This is a schematic diagram of a BEV adaptive online mapping system provided in an embodiment of the present invention.

[0017] Figure 5 This is a schematic diagram of a processing device provided in an embodiment of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0019] First, the following explanations are provided for the terms that may be used in this article: The terms "comprising," "including," "containing," "having," or other similar semantic descriptions should be interpreted as non-exclusive inclusion. For example, including a technical feature element (such as raw material, component, ingredient, carrier, dosage form, material, size, part, component, mechanism, device, step, process, method, reaction conditions, processing conditions, parameter, algorithm, signal, data, product or article of manufacture, etc.) should be interpreted as including not only the expressly listed technical feature element, but also other technical feature elements that are not expressly listed and are well-known in the art.

[0020] The term "composed of" excludes any technical features not expressly listed. When used in a claim, it closes the claim to exclude all technical features other than those expressly listed, except for associated conventional impurities. If the term appears only in a clause of a claim, it limits the claim to the elements expressly listed in that clause; elements recited in other clauses are not excluded from the overall claim.

[0021] The following provides a detailed description of a BEV adaptive online mapping method, system, device, and storage medium provided by this invention. Contents not described in detail in the embodiments of this invention are prior art known to those skilled in the art. Where specific conditions are not specified in the embodiments of this invention, they are performed according to conventional conditions in the art or conditions recommended by the manufacturer. Instruments used in the embodiments of this invention, unless otherwise specified by the manufacturer, are all commercially available conventional products.

[0022] Example 1 This invention provides a BEV adaptive online mapping method, such as... Figure 1 As shown, it mainly includes the following steps: Step 1: Generation of distortion-aware BEV features.

[0023] In this embodiment of the invention, feature extraction is performed on the input images from the pinhole camera and the surround-view fisheye camera to obtain corresponding feature maps; the initial BEV reference point is perturbed by an adaptive offset strategy, and the two-dimensional reference point corresponding to each image is determined based on the perturbed BEV reference point; the corresponding feature map is weighted based on the two-dimensional reference point to obtain a weighted feature map; and the weighted feature maps corresponding to all images are aggregated into BEV features by combining an attention mechanism.

[0024] The following details the relevant aspects of this step:

[0025] (1) The initial BEV reference point is perturbed by an adaptive offset strategy, and the two-dimensional reference point corresponding to each image is determined based on the perturbed BEV reference point.

[0026] In this embodiment of the invention, the set of three-dimensional initial BEV reference points uniformly distributed in the BEV space is denoted as... ,in, This represents the position coordinates of the k-th initial BEV reference point, where K is the number of BEV reference points.

[0027] for Introducing dynamic offset Obtain the BEV reference point after disturbance. : ; in, Generate using the given BEV query sequence Q;

[0028] Using the perturbated BEV reference point The two-dimensional reference point corresponding to each image is found in the lookup table; wherein, the lookup table stores the mapping relationship between each initial BEV reference point and the two-dimensional reference point corresponding to each image.

[0029] (2) Calculate the mapping relationship between all BEV reference points and the corresponding two-dimensional reference points of each image offline in advance, and construct a lookup table. The lookup table can be used to find the two-dimensional reference points of each initial BEV reference point and each image.

[0030] In this embodiment of the invention, the coordinates of the kth initial BEV reference point are denoted as... The two-dimensional reference point obtained by the following formula : ; Where i corresponds to the i-th image. Let i be the projection function of the camera corresponding to the i-th image, containing the intrinsic parameter matrix. extrinsic parameter matrix and camera distortion model Substituting into the above equation, we get: .

[0031] Among them, camera distortion model Through polynomial model parameterization, the pinhole camera can be represented by the identity mapping as follows: .

[0032] For a panoramic fisheye camera, the imaging process exhibits significant nonlinear characteristics. The distortion model is based on polynomial parameterization of the light incident angle, specifically expressed as follows: ; in: , , ; Let (u, v) be the distortion model of the i-th camera, and (u, v) be the distortion-free normalized image plane coordinates. These are the normalized coordinates of the distortion, where r represents the distance from point (u, v) to the image center. It is the undistorted angle of incidence. It is the distorted angle of incidence. , , and These are the distortion parameters for the distortion model of the panoramic fisheye camera.

[0033] (3) Perform bilinear interpolation sampling on the feature map of the corresponding viewpoint based on the two-dimensional reference point to obtain the weighted feature map. .

[0034] Step 2: Lane modeling based on adaptive lane attention.

[0035] In this embodiment of the invention, the semantic association of the learnable map instance query representation corresponding to each lane line segment is modeled by a self-attention mechanism to obtain the updated map instance query representation of each lane line segment; for each lane line segment, the sampling weight of each reference point is calculated based on the distance relationship between its corresponding reference points, and the updated map instance query representation and BEV features are combined to obtain the corresponding feature response through the attention mechanism.

[0036] The following details the relevant aspects of this step: (1) For each lane line segment, the sampling weight of each reference point is calculated based on the distance relationship between its corresponding reference points.

[0037] Each lane line segment corresponds to L reference points, and the set of reference points is denoted as L. ,in, This indicates the nth lane line segment. One reference point; The nth lane line segment is determined by the following formula. Sampling weights of each reference point : ; in, For the nth lane line segment A reference point, It is the Sigmoid activation function. It is a single-layer MLP. Query the map instance representation after updating the nth lane line segment.

[0038] (2) Combining the updated map instance query representation with BEV features, the corresponding feature response is obtained through an attention mechanism, represented as: ; in, Characteristics of BEV This represents a deformable attention operation based on the reference point position; This indicates the nth lane line segment. A reference point, for Sampling weights, Query the map instance representation after updating the nth lane line segment. This represents the feature response corresponding to the nth lane line segment.

[0039] (3) Use the feature vector of each lane line segment for classification and regression to obtain the lane category and point cloud in BEV space for each lane line segment.

[0040] Step 3: Construct a complete BEV lane topology map.

[0041] In this embodiment of the invention, a complete BEV lane topology map is constructed by combining the connection relationships between lane line segments, the lane category of each lane line segment, and the point cloud in the BEV space.

[0042] In this embodiment of the invention, the connection relationship between lane line segments is obtained by using the similarity between the updated map instance query representations of each lane line segment to obtain a binarized adjacency matrix. Each element of the adjacency matrix represents the connection relationship between the corresponding lane line segments.

[0043] In this embodiment of the invention, step 1 is implemented by an encoding module, and step 2 is implemented by a decoding module. They are trained using the following loss function: ; in, The loss function; For classification loss, it is calculated using the difference between the predicted lane category and the true category for each lane line segment; The Manhattan distance loss is calculated using the Manhattan distance between neighboring points in the point cloud within the predicted BEV space. For topological loss, a binarized adjacency matrix is ​​used. The binarized adjacency matrix is ​​calculated by comparing the differences in connectivity between the actual lane line segments and the actual lane line segments. This is obtained by querying the similarity between the map instances updated for each lane line segment; To calculate the rendering loss, the difference between the smooth curve obtained by interpolating the point cloud in the predicted BEV space and the actual lane line is used. and These are hyperparameters used to control the weights of the corresponding loss.

[0044] The above-mentioned solution provided by the embodiments of the present invention starts from the joint perception of surrounding pinhole and fisheye images, uses multi-view pinhole and fisheye images to extract image features, considers the distortion models of pinhole and fisheye cameras, adaptively maps and generates BEV features, and designs adaptive lane attention for BEV online mapping tasks to solve the joint perception difficulties caused by fisheye image distortion, thereby improving the prediction accuracy and stability of online mapping.

[0045] To more clearly demonstrate the technical solution and its effects provided by the present invention, the method provided by the embodiments of the present invention will be described in detail below with reference to specific examples.

[0046] I. Overall Overview of the Plan

[0047] To address the problems existing in current BEV online mapping technologies based on surround-view pinhole and fisheye cameras, such as oversensitivity to camera calibration parameters, difficulty in effectively handling severe radial distortion, perception distortion in dynamic scenes, and reliance on high-cost, high-precision map priors, this invention provides an adaptive online mapping method for BEVs based on joint perception using surround-view pinhole and fisheye cameras. Furthermore, existing BEV mapping schemes are mainly based on multi-view pinhole cameras and rely on precise geometric models and calibrations, resulting in poor robustness to parameter changes and dynamic objects; or they employ complex models leading to high computational costs, making them difficult to adapt to resource-constrained in-vehicle platforms; and they generally lack targeted optimization for the distortion characteristics of fisheye images, resulting in a significant decrease in edge region perception performance.

[0048] In the field of autonomous driving, perception is a core component of autonomous driving systems. BEV online mapping technology, as an important part of the perception module, can construct lane topology results from the BEV's perspective in real time, providing data for downstream planning. In many scenarios, fisheye camera systems are widely used due to their ultra-large field of view and blind-spot-free characteristics. However, the edge compression distortion of fisheye images poses a significant challenge to BEV perception tasks, especially BEV online mapping tasks.

[0049] To address this, the present invention provides a distortion-aware adaptive BEV module that adaptively models the non-uniform distortion characteristics of fisheye images, achieving accurate mapping of image features to the BEV space without relying on absolute calibration precision. Furthermore, by designing an adaptive lane attention mechanism, the accuracy of static map element construction is ensured. This method significantly improves the accuracy and reliability of lane topology perception in autonomous driving systems based on low-cost fisheye cameras under complex real-world street scenes, providing a solid technical guarantee for downstream path planning and control decisions.

[0050] II. Detailed introduction of the plan.

[0051] This invention presents an innovative design for online BEV mapping methods using surround-view pinhole and fisheye camera systems, establishing a novel distortion-aware online BEV mapping framework for surround-view pinhole and fisheye camera systems. First, it proposes an adaptive BEV module for distortion awareness, leveraging the large field of view (FOV) of the surround-view fisheye camera for joint perception. The feature mapping module is optimized based on the fisheye camera distortion model, and an adaptive offset is incorporated into the reference point sampling to enhance the model's robustness. This enables efficient mapping from surround-view distortion images to dense BEV features while conserving computational resources. Second, this invention proposes adaptive lane attention, allowing different map instance queries to adaptively interact based on lane-specific structural priors, thereby improving the accuracy of lane modeling.

[0052] like Figure 2 The diagram shows the overall architecture of the method described above in this invention. Based on the Encoder-Decoder structure, the Encoder part takes the panoramic image as input, encodes the image features, and maps them to the BEV perspective to generate BEV features. The Decoder part is a decoder for map instances, which interacts with the BEV features using the initialized map instance query, and finally obtains the map prediction through classification and regression.

[0053] 1. Encoder section.

[0054] This invention employs a unified two-dimensional backbone network to process images from pinhole cameras and surround-view fisheye cameras. Feature extraction is performed separately to obtain high-resolution feature maps from each viewpoint. Where C is the number of channels. , R represents the resolution after downsampling, and R is the set of real numbers. This backbone network can be a ResNe50 architecture or can be replaced with other CNN or ViT architectures. Its output features retain rich spatial detail information and are suitable for subsequent multi-view feature fusion.

[0055] For example, the number of pinhole cameras and surround-view fisheye cameras can be set to 2 each, resulting in a total of 4 input images.

[0056] Next, in the BEV feature construction stage, this invention designs a distortion-aware adaptive BEV module to achieve dense feature mapping directly from multi-view 2D features to the BEV space, and adapts accordingly to the distortion characteristics of fisheye, such as... Figure 3 The diagram illustrates the principle of the adaptive BEV module. Let the initial three-dimensional set of BEV reference points be uniformly distributed in the BEV space. ,in This represents the position coordinates of the k-th initial BEV reference point. These are the coordinate values ​​on the x, y, and z axes (the z-direction typically takes H values ​​at a fixed height, where H=4). For each... By projecting it onto the image plane of each viewpoint through inverse perspective transformation, the corresponding 2D reference points are obtained. ,Right now: in Let i be the projection function of the camera corresponding to the i-th image, containing the intrinsic parameter matrix. extrinsic parameter matrix and camera distortion model Because fisheye lenses exhibit significant nonlinear distortion, traditional linear projection cannot accurately reflect the true geometric relationships. Therefore, this invention explicitly models the distortion term during the projection process: .

[0057] Among them, camera distortion model Through polynomial model parameterization, the pinhole camera can be represented by the identity mapping as follows: .

[0058] For fisheye cameras, the imaging process exhibits significant nonlinear characteristics. The distortion model is based on polynomial parameterization of the light incident angle, specifically expressed as follows: ; in: , , , Let (u, v) be the distortion model of the i-th camera, and (u, v) be the distortion-free normalized image plane coordinates. These are the normalized coordinates of the distortion, where r represents the distance from point (u, v) to the image center. It is the angle of incidence. It is the distorted angle of incidence. , , and These are the distortion parameters for the distortion model of the panoramic fisheye camera.

[0059] To improve computational efficiency, this invention pre-calculates all offline calculations. The mapping relationship is established, and a global lookup table (LUT) T is constructed to store the 2D projected coordinates of each BEV reference point under all viewpoints and its corresponding pixel neighborhood index. During training, the projection position is directly obtained by looking up the table, avoiding real-time calculation of complex distortion functions, thereby significantly reducing inference latency. To further enhance the model's robustness to edge regions of fisheye images and reduce the model's sensitivity to external parameter errors, this invention proposes an adaptive offset strategy, which is applied to the initial BEV reference point. Introducing dynamic offset The reference point is formed after the disturbance: ; offset Composed of a lightweight, learnable module Generated based on the current BEV query sequence Q. This offset mechanism allows the model to dynamically adjust the sampling position according to the actual scene, especially in areas with severe distortion such as road edges and occluded areas, effectively alleviating the feature loss problem caused by fixed sampling. After obtaining the perturbation reference point... Then, the two-dimensional reference points for each viewpoint are obtained by looking up table T. and in the feature map of the corresponding viewpoint Perform bilinear interpolation sampling to obtain weighted features. .

[0060] In this embodiment of the invention, since the original fisheye image captured by the fisheye camera has invalid regions with black edges, a fixed Valid Mask (valid region mask) is pre-marked for each fisheye camera image. This operation is pre-completed, with one fixed valid mask corresponding to one fisheye camera. This Valid Mask is used when sampling image features from two-dimensional reference points, that is, the sampling features of points falling within the Valid Mask are valid, and those falling outside the Valid Mask are invalid.

[0061] In this embodiment of the invention, when creating the lookup table, a voxel center point of a specified size in the BEV space is used as an initial BEV reference point. Then, the corresponding two-dimensional reference point can be determined based on the initial BEV reference point corresponding to the voxel where the perturbed reference point is located. For example, a voxel center point of 0.1m × 0.1m × 0.1m in the BEV space can be used as an initial BEV reference point. The projection relationships from these 3D reference points to the four cameras are calculated for all voxels within 15m to the left and right and 30m to the front and back in the BEV space.

[0062] Finally, the sampled features from all perspectives are fused to complete the BEV feature aggregation: ; in, For the cross-attention mechanism, O is the number of images (e.g., O=4 in the previous example), and Q is the BEV query sequence. This represents the sampling result of the k-th reference point from the i-th viewpoint. This process achieves sparse and efficient aggregation of multi-view features while taking into account the geometric distortion caused by fisheye distortion. In summary, this invention significantly improves the accuracy and robustness of BEV feature generation without increasing additional computational overhead by constructing a distortion-aware reference point mapping mechanism and an adaptive offset strategy, especially maintaining good mapping performance under extreme distortion conditions.

[0063] 2. Decoder section.

[0064] Based on the obtained BEV characteristics This invention employs a lane segment modeling approach, decomposing the road structure into several independent lane segments, each represented by a learnable map instance query. This map instance query is used to perceive and reconstruct the corresponding lane geometry in the BEV space, and infers lane connectivity relationships through the interaction between map instance query representations.

[0065] Specifically, suppose there are N lane line segments, and the corresponding map instance query representation is denoted as . Where d is the query vector dimension. Initially, map instance query representations are randomly initialized or pre-assigned location information through a clustering strategy. During the decoding phase, a self-attention mechanism is first performed on all queries to model the semantic relationships between different lane line segments: ; in, This is a map instance query representation of the m-th lane line segment.

[0066] Subsequently, each updated map instance query represents Used with BEV characteristics Interact with the system to obtain spatial context information of the corresponding lane line segment. Specifically, this invention provides a method for each query... Initialize a set of reference points, distributed based on the typical geometry of lane line segments. For example, L=12 reference points can be set, with 6 reference points evenly spaced on the left lane line and 6 reference points evenly spaced on the right lane line, forming a symmetrical layout. Let the set of reference points for the nth query be... Its coordinates in the BEV plane are generated as follows: ; Among them, the left point Evenly distributed along the left boundary line, points on the right side ( The points are uniformly distributed along the right boundary line with a fixed distance between them to simulate real lane changes. To enhance sampling robustness, this invention introduces an adaptive learnable weight mechanism. For each reference point... Its sampling weights on BEV features It's not fixed, but rather through a lightweight network. Dynamically generated, the network input is determined by the average relative distance between the reference point and the other L-1 reference points in the uniform lane line segment, and the updated map instance query representation. ; in, For the nth lane line segment A reference point, The Sigmoid activation function ensures weight normalization; lightweight network. It can be a single-layer MLP, and the parameters are trainable; The map instance is queried after the nth lane line segment is updated. This mechanism allows the model to automatically adjust sampling weights based on local geometric complexity, such as giving higher weights to edge points at curves.

[0067] Then, using the aforementioned weighted reference points... Deformable attention sampling is performed on the sample to obtain the corresponding feature response: ; in, This represents a deformable attention operation based on the reference point location, which extracts multi-scale contextual information from the BEV feature map.

[0068] Based on the obtained feature response The classification head and regression head are used to process the lane line segments and output their categories. With geometric point sets : Classification header output: ; Return head output: ; in, For classification header, The number of lane types (combinations of solid lines, dashed lines, double yellow lines, etc.). M is the regression head, and M is the number of points regressed for each lane line segment. The regression result is the point cloud representation of the lane line segment in the BEV space.

[0069] 3. Construct a complete BEV lane topology map.

[0070] To establish topological connections between lanes, this invention further designs an adjacency matrix prediction module. First, the similarity between the updated map instance query representations of any two lane line segments is calculated to construct candidate connections: ; in, Query the similarity of the representations of the nth and mth lane line segments after their updates to the map instance. It is a similarity-independent function. Let T be the map instance query representation updated for the m-th lane line segment, where T is the transpose symbol.

[0071] The similarity matrix can be obtained through the above method. Then, it is input into a small classification network, which outputs a binarized adjacency matrix. Each element at a given location indicates whether there is a connection between lane line segments.

[0072] In this embodiment of the invention, a complete BEV lane topology map is constructed by combining the connection relationships between lane line segments, the lane category of each lane line segment, and the point cloud in the BEV space.

[0073] 4. Training loss.

[0074] The overall loss function of this invention consists of multiple sub-items, covering three aspects: geometric accuracy, semantic consistency, and topological structure, and is used to jointly train the above-mentioned Encoder and Decoder parts.

[0075] (1) Classification loss: For class prediction of each lane line segment, cross-entropy loss is used: ; in, Let n be the true type of the nth lane line segment.

[0076] (2) Manhattan distance loss: used to constrain the geometric smoothness of the regression point set. Defined as the sum of the Manhattan distances between adjacent points: ; in, and geometric point set The j-th and j+1-th points in the array.

[0077] (3) Topological loss: used to optimize adjacency matrix prediction, employing binary cross-entropy loss: ; in, This represents the connection relationship between actual lane line segments. The binary adjacency matrix is ​​calculated (the calculation method can be found in the previous introduction).

[0078] (4) Rendering loss: The predicted point set is interpolated using B-spline to obtain a smooth curve. and with GT lane lines Perform pixel-level matching and calculate the binary cross-entropy loss: ; Wherein, BCE is the binary cross-entropy function.

[0079] The final total loss is the weighted sum of all items: ; in, It is an adjustable hyperparameter.

[0080] Since the process of subsequent training based on the above losses can be implemented by referring to conventional techniques, it will not be elaborated further.

[0081] The above-mentioned solution provided by the present invention has the following main advantages: (1) The distortion-aware BEV feature generation mechanism significantly improves mapping accuracy and edge robustness. This mechanism abandons the simplified processing of assuming a pinhole camera model and the cumbersome process of first removing distortion and then sensing in fisheye cameras in traditional methods. It explicitly introduces the radial distortion model of the fisheye camera in the multi-view BEV feature aggregation process and achieves efficient projection mapping through pre-computed lookup tables. At the same time, an innovative adaptive reference point offset strategy is designed, which allows the model to dynamically adjust the sampling position according to the local context, and can still stably extract effective features, especially in areas with severe image edge distortion. This fundamentally solves the problem of BEV spatial feature distortion caused by nonlinear compression distortion of fisheye images, and does not highly depend on the accuracy of camera extrinsic parameters. Without increasing the additional computational burden, it greatly improves the geometric consistency and semantic integrity of BEV features, providing a high-fidelity and high-stability feature foundation for downstream lane modeling.

[0082] (2) The adaptive lane line segment reference point sampling mechanism achieves high-precision geometric reconstruction and structured representation. This invention breaks through the traditional mapping paradigm of pixel-by-pixel segmentation or dense regression, and adopts an instantiation modeling method with lane line segments as the basic unit. Each lane line segment is represented by an independent query and initialized with 12 reference points that conform to the real lane geometry distribution. Combined with a learnable adaptive weight mechanism, the model can dynamically adjust the sampling strategy according to local curvature and structural complexity, so as to maintain the continuity and smoothness of lane line shape in complex scenarios such as curves, bifurcations, and occlusions. This design not only improves the accuracy of lane geometry modeling, but also makes the output naturally have structured semantics (such as lane ID, type, point sequence), which can directly serve the planning and control module, avoiding the tedious clustering and fitting steps in post-processing, and greatly improving the efficiency of system engineering.

[0083] In summary, this invention first constructs a distortion-perceived BEV feature generation framework for surround-view pinhole and fisheye cameras, balancing efficiency and robustness through lookup table acceleration and adaptive offset strategies. Secondly, it designs a lane line segment-level instantiation modeling and structural perception reference point sampling mechanism, and incorporates adaptive attention-weighted reference points to achieve high-precision geometric reconstruction. Finally, it introduces an end-to-end topology prediction module based on query similarity, completing the closed loop from geometric perception to semantic association. These progressive and synergistic processes collectively construct a high-precision, robust, structured, end-to-end online mapping system for BEVs using combined surround-view pinhole and fisheye camera perception. This effectively addresses the core pain points of existing technologies in distortion adaptability, geometric modeling accuracy, and topological integrity, providing robust and reliable online map support for the integrated perception-planning architecture of autonomous driving.

[0084] III. Specific Examples.

[0085] Based on the above-described scheme, this example uses an Encoder-Decoder architecture as its core. It adapts to the non-uniform distortion characteristics of fisheye images through a distortion-aware adaptive BEV module, and combines this with an adaptive lane attention module to achieve accurate lane topology modeling. The final output is a real-time BEV map that meets the planning requirements of autonomous driving. First, distorted images are acquired simultaneously from two front- and rear-view pinhole cameras and two surround-view fisheye cameras. Dense BEV features are obtained through feature extraction and BEV space mapping. Then, through map instance decoding and topology prediction, the lane segment categories, geometric shapes, and connectivity relationships are output.

[0086] Step S1: Prepare the training dataset. Images captured simultaneously by four cameras should form a group. The dataset must cover multiple scenes, including urban roads, highways, and construction zones, and include data under different lighting conditions (sunny days, nighttime) and weather conditions (rainy days, foggy days). Preprocess each image: uniformly scale it to a preset resolution (e.g., 1280×720) to preserve distorted edges; normalize pixel values ​​(mapping pixel values ​​to the [0,1] range); and enhance the image using random horizontal flipping, brightness / contrast perturbation (±20%), and Gaussian noise addition (standard deviation ≤0.05). Avoid random cropping to prevent damage to the integrity of distorted areas. The final dataset must include grouped panoramic images and corresponding BEV spatial annotations (lane type, geometric coordinates, topological connectivity), and be divided into training, validation, and test sets in a 7:2:1 ratio.

[0087] Step S2: An online mapping model for surround-view fisheye BEV is built using the deep learning framework PyTorch. The model consists of two parts: an Encoder and a Decoder. The Encoder includes a 2D backbone network and a distortion-aware adaptive BEV module: the 2D backbone network uses ResNet50, removing the fully connected layers at the ends, outputting multi-scale feature maps, and unifying the number of channels through 1×1 convolutions to obtain high-resolution feature maps for each viewpoint; the distortion-aware adaptive BEV module needs to pre-compute a global lookup table (LUT) to store the 2D projection coordinates and pixel neighborhood indices of uniformly distributed 3D reference points in the BEV space on the image plane of each viewpoint, and also designs a lightweight learnable module to generate adaptive offsets for the reference points. The Decoder includes an adaptive lane attention module, a classification regression head, and a topology prediction module: the adaptive lane attention module initializes lane segment queries and models the association between queries through self-attention, configuring 12 reference points conforming to lane geometry and learnable sampling weights for each query; the classification regression head outputs the lane segment category and geometric point set, and the topology prediction module constructs an adjacency matrix based on query similarity.

[0088] Step S3: Build a four-way surround-view camera system. Install the front and rear pinhole cameras and the left and right fisheye cameras on the test vehicle. Ensure that the overlap rate of the fields of view of adjacent cameras is ≥15%. Use Zhang Zhengyou calibration method combined with fisheye distortion model to obtain the camera intrinsic parameters (focal length, optical center, radial distortion coefficient) and extrinsic parameters (rotation matrix, translation matrix). Correct the extrinsic parameter error to ensure projection accuracy. Set the camera sampling frame rate to 15fps to ensure that the time synchronization error of multi-view images is ≤50ms.

[0089] Step S4: Input the training ensemble group of panoramic images into the 2D backbone network of the Encoder to obtain high-resolution feature maps of four perspectives. At this time, process all perspective feature maps as a unified batch and use GPU parallel computing to improve feature extraction efficiency.

[0090] Step S5: Based on the pre-computed LUT, project the 3D reference points of the BEV space onto the image planes of each viewpoint to obtain the initial 2D reference points; add dynamic offsets to the 3D reference points through the adaptive offset module to generate perturbed reference points and project them again to obtain new 2D reference points; perform bilinear interpolation sampling on the corresponding viewpoint feature maps to obtain weighted features; and finally fuse the features of the four viewpoints through the deformable attention module to output a dense BEV feature map.

[0091] Step S6: Enter the Decoder stage, initialize N lane segment queries (e.g., N=30), update the queries through a self-attention mechanism to model the semantic association between lane segments; initialize 12 reference points for each updated query (6 on the left and 6 on the right, with a 1m spacing between adjacent points), generate sampling weights for each reference point through a lightweight network, and perform deformable attention sampling on the BEV feature map based on the reference point positions and weights to obtain the feature response for each query.

[0092] Step S7: Input the query feature response into the classification head and regression head: The classification head outputs the lane segment category probability (e.g., distinguishing between solid lines, dashed lines, double yellow lines, etc.), and the regression head outputs the geometric point set of the lane segment in the BEV space (e.g., regressing 20 points for each lane segment); calculate the cosine similarity between any two queries, input it into the binary classification network to obtain the binary adjacency matrix, which represents the connection relationship between lane segments.

[0093] Step S8: Calculate the total model loss, which includes classification loss (cross-entropy loss, constraining the accuracy of class prediction), Manhattan distance loss (constraining the smoothness of lane geometry point set), topological loss (binary cross-entropy loss, optimizing adjacency matrix prediction), and rendering loss (interpolating the predicted lane lines to smooth curves and matching them with the ground plane at the pixel level, calculating the binary cross-entropy loss). The total loss is the weighted sum of the various losses (e.g., hyperparameters α=0.5, β=1.0, γ=0.8).

[0094] Step S9: Using the AdamW optimizer (initial learning rate 1e-4, weight decay 1e-5) and gradient descent strategy, minimize the total loss through backpropagation and update the weights of each module of the model. During training, warm up the model with the learning rate for the first 10 rounds, and decay the learning rate with the cosine annealing strategy after the 50th round. Evaluate the model performance on the validation set every 2 epochs (using lane geometry error and topology accuracy as indicators) and save the optimal model weights.

[0095] Step S10: Input the test set group surround view image into the trained model, execute the inference process of steps S4-S7, output the lane segment category, geometric coordinates and topological adjacency matrix, and construct a complete BEV lane topology map; calculate the model performance indicators on the test set (such as lane geometric error ≤0.3m, category accuracy ≥95%, topology accuracy ≥90%), and verify the effectiveness of the method.

[0096] Through the above description of the embodiments, those skilled in the art can clearly understand that the above embodiments can be implemented by software, or by using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions of the above embodiments can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, mobile hard drive, etc.), including several instructions to cause a computer device (such as a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0097] Example 2 This invention also provides a BEV adaptive online mapping system, which is mainly used to implement the methods provided in the foregoing embodiments, such as... Figure 4 As shown, the system mainly includes: The encoding module is used to extract features from the input images from the pinhole camera and the surround-view fisheye camera to obtain the corresponding feature maps; the BEV reference points are perturbed by an adaptive offset strategy, and the two-dimensional reference points corresponding to each image are determined based on the perturbed BEV reference points; the corresponding feature maps are weighted based on the two-dimensional reference points to obtain weighted feature maps; and the weighted feature maps corresponding to all images are aggregated into BEV features by combining an attention mechanism. The decoding module models the semantic relationships between the learnable map instance query representations corresponding to each lane line segment using a self-attention mechanism, obtaining the updated map instance query representations for each lane line segment. For each lane line segment, it calculates the sampling weights of each reference point based on the distance relationships between their corresponding reference points, and combines the updated map instance query representation with BEV features to obtain the corresponding feature response through an attention mechanism. Finally, it uses the feature vectors of each lane line segment for classification and regression to obtain the lane category and point cloud in the BEV space for each lane line segment. The mapping module is used to construct a complete BEV lane topology map by combining the connection relationships between lane line segments, the lane category of each lane line segment, and the point cloud in the BEV space.

[0098] Since the main technical details of this system have been described in detail in previous embodiments, they will not be repeated here.

[0099] Those skilled in the art will understand that, for the sake of convenience and brevity, the above-described division of functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0100] Example 3 The present invention also provides a processing device, such as Figure 5 As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided in the foregoing embodiments.

[0101] Furthermore, the processing device also includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.

[0102] In this embodiment of the invention, the specific types of the memory, input device, and output device are not limited; for example: Input devices can be touchscreens, image acquisition devices, physical buttons, or mice, etc. The output device can be a display terminal; The memory can be random access memory (RAM) or non-volatile memory, such as disk storage.

[0103] Example 4 The present invention also provides a readable storage medium storing a computer program that, when executed by a processor, implements the method provided in the foregoing embodiments.

[0104] In this embodiment of the invention, the readable storage medium is a computer-readable storage medium and can be disposed in the aforementioned processing device, for example, as a memory in the processing device. Furthermore, the readable storage medium can also be any medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0105] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims. The information disclosed in the background section is intended only to enhance the understanding of the overall background technology of the present invention and should not be construed as an admission or implication in any way that such information constitutes prior art known to those skilled in the art.

Claims

1. A BEV adaptive online mapping method, characterized in that, include: Step 1: Extract features from the input pinhole camera and surround-view fisheye camera images to obtain corresponding feature maps; perturb the BEV reference points using an adaptive offset strategy, and determine the corresponding two-dimensional reference points for each image based on the perturbated BEV reference points; weight the corresponding feature maps based on the two-dimensional reference points to obtain weighted feature maps; combine the attention mechanism to aggregate the weighted feature maps corresponding to all images into BEV features. Step 2: Model the semantic associations of the learnable map instance query representations corresponding to each lane line segment through a self-attention mechanism to obtain the updated map instance query representations for each lane line segment; for each lane line segment, calculate the sampling weights of each reference point based on the distance relationship between its corresponding reference points, and combine the updated map instance query representations with BEV features to obtain the corresponding feature responses through an attention mechanism; use the feature vectors of each lane line segment for classification and regression to obtain the lane category and point cloud in the BEV space for each lane line segment; Step 3: Combine the connection relationships between lane line segments, the lane category of each lane line segment, and the point cloud in the BEV space to construct a complete BEV lane topology map.

2. The BEV adaptive online mapping method according to claim 1, characterized in that, The step of perturbing the BEV reference points using an adaptive offset strategy and determining the corresponding two-dimensional reference points for each image based on the perturbed BEV reference points includes: Let the set of three-dimensional BEV reference points uniformly distributed in the BEV space be denoted as... ,in, This represents the position coordinates of the kth BEV reference point, where K is the number of BEV reference points; for Introducing dynamic offset Obtain the BEV reference point after disturbance. : ； in, Generate using a given BEV query; Using the perturbated BEV reference point The two-dimensional reference point corresponding to each image is found in the lookup table; wherein, the lookup table stores the mapping relationship between each BEV reference point and the two-dimensional reference point corresponding to each image.

3. The BEV adaptive online mapping method according to claim 1, characterized in that, The method also includes: pre-calculating the mapping relationship between all BEV reference points and the corresponding two-dimensional reference points of each image offline, and constructing a lookup table, through which the two-dimensional reference points of each BEV reference point and each image can be found; Let the coordinates of the kth BEV reference point be denoted as The two-dimensional reference point obtained by the following formula : ； Where i corresponds to the i-th image. Let i be the projection function of the camera corresponding to the i-th image, containing the intrinsic parameter matrix. extrinsic parameter matrix and camera distortion model Substituting into the above equation, we get: ； Among them, camera distortion model Parameterization is achieved through a polynomial model.

4. The BEV adaptive online mapping method according to claim 3, characterized in that, The camera distortion model Parameterization through polynomial models includes: For a pinhole camera, the identity mapping is represented as: ； For a panoramic fisheye camera, it is represented as: ； in: , , ; Let (u, v) be the distortion model of the i-th camera, and (u, v) be the distortion-free normalized image plane coordinates. These are the normalized coordinates of the distortion, where r represents the distance from point (u, v) to the image center. It is the undistorted angle of incidence. It is the distorted angle of incidence. , , and These are the distortion parameters for the distortion model of the panoramic fisheye camera.

5. The BEV adaptive online mapping method according to claim 1, characterized in that, The calculation of the sampling weights of each reference point for each lane line segment, based on the distance relationship between its corresponding reference points, includes: Each lane line segment corresponds to L reference points, and the set of reference points is denoted as L. ,in, This indicates the nth lane line segment. One reference point; The nth lane line segment is determined by the following formula. Sampling weights of each reference point : ； in, For the nth lane line segment A reference point, It is the Sigmoid activation function. It is a single-layer MLP. Query the map instance representation after updating the nth lane line segment.

6. The BEV adaptive online mapping method according to claim 1, characterized in that, The updated map instance query representation, combined with BEV features, is used to obtain the corresponding feature response representation through an attention mechanism, as follows: ； in, Characteristics of BEV This represents a deformable attention operation based on the reference point position; This indicates the nth lane line segment. A reference point, for Sampling weights, Query the map instance representation after updating the nth lane line segment. This represents the feature response corresponding to the nth lane line segment.

7. The BEV adaptive online mapping method according to claim 1, characterized in that, Step 1 is implemented through the encoding module, and step 2 is implemented through the decoding module. They are trained using the following loss function: ； in, The loss function; For classification loss, it is calculated using the difference between the predicted lane category and the true category for each lane line segment; The Manhattan distance loss is calculated using the Manhattan distance between neighboring points in the point cloud within the predicted BEV space. For topological loss, a binarized adjacency matrix is ​​used. The binarized adjacency matrix is ​​calculated by comparing the differences in connectivity between the actual lane line segments and the actual lane line segments. This is obtained by querying the similarity between the map instances updated for each lane line segment; To calculate the rendering loss, the difference between the smooth curve obtained by interpolating the point cloud in the predicted BEV space and the actual lane line is used. and These are hyperparameters used to control the weights of the corresponding loss.

8. A BEV adaptive online mapping system, characterized in that, To implement the method according to any one of claims 1 to 7, comprising: The encoding module is used to extract features from the input images from the pinhole camera and the surround-view fisheye camera to obtain the corresponding feature maps; the BEV reference points are perturbed by an adaptive offset strategy, and the two-dimensional reference points corresponding to each image are determined based on the perturbed BEV reference points; the corresponding feature maps are weighted based on the two-dimensional reference points to obtain weighted feature maps; and the weighted feature maps corresponding to all images are aggregated into BEV features by combining an attention mechanism. The decoding module is used to model the semantic associations of the learnable map instance query representations corresponding to each lane line segment through a self-attention mechanism, thereby obtaining the updated map instance query representations for each lane line segment. For each lane line segment, the sampling weights of each reference point are calculated based on the distance relationship between its corresponding reference points, and the corresponding feature responses are obtained through an attention mechanism by combining the updated map instance query representations with BEV features. The feature vectors of each lane line segment are used for classification and regression to obtain the lane category and point cloud in the BEV space for each lane line segment. The mapping module is used to construct a complete BEV lane topology map by combining the connection relationships between lane line segments, the lane category of each lane line segment, and the point cloud in the BEV space.

9. A processing device, characterized in that, include: One or more processors; Memory, used to store one or more programs; Wherein, when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method as described in any one of claims 1 to 7.

10. A readable storage medium storing a computer program, characterized in that, When a computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

Images