Method and apparatus for generating road map based on end-to-end visual neural network

Abstract

The application discloses a method and device for generating a road map based on an end-to-end visual neural network, and belongs to the technical fields of automatic driving, computer vision and high-precision map construction; the device comprises an input layer, a perception and fusion front end, a visual space conversion and BEV bird's-eye view fusion module, a geometric and topological joint decoding back end, an output and application layer; a road geometric and topological map is generated based on multi-camera fused image features in an end-to-end mode, a single end-to-end road geometric and topological map generation model is constructed and trained, and map elements are directly output from multi-view images. The application breaks through the vehicle-mounted visual angle limitation, greatly improves the map generation integrity and anti-occlusion capability, achieves the laser radar precision in a pure visual scheme, has better robustness under poor environment and complex weather conditions, and can solve the technical problems of high production cost, poor updating timeliness and separated geometric and topological generation processes in the prior art.

Description

Technical Field

[0001] This invention belongs to the fields of autonomous driving, computer vision and high-precision map construction technology. It relates to a technology for road environment modeling using roadside perception devices, and more particularly to a method and apparatus for end-to-end generation of vectorized road maps based on deep learning neural networks. The generated map contains two core types of information: road geometry and topological relationships. Background Technology

[0002] High-definition maps (HD maps) are a critical infrastructure for the safe and reliable operation of Level 3 and above autonomous driving systems. HD maps not only need to contain centimeter-level precise spatial geometric information of road elements such as lane lines, stop lines, and pedestrian crossings, but also must define the logical connection relationships between these elements, i.e., road topology (e.g., merging, diverging, or turning relationships between lanes).

[0003] Currently, the mainstream technology for producing high-precision maps relies on specialized map data collection vehicles. These vehicles are typically equipped with high-cost LiDAR, high-precision GPS, and inertial measurement units (IMUs). Data collection teams acquire massive amounts of 3D point cloud data through on-site road surveys, and then professionals generate maps using complex offline data processing procedures (such as point cloud registration, feature extraction, and manual annotation and correction). The industry is beginning to explore lower-cost visual solutions, such as utilizing widely deployed urban roadside cameras.

[0004] Existing solutions and mainstream technical approaches suffer from the following drawbacks and technical bottlenecks:

[0005] High production and maintenance costs: Professional sensor hardware such as LiDAR is expensive, and data acquisition and subsequent manual processing require huge capital and time investment, resulting in extremely high initial construction costs and subsequent maintenance costs for high-precision maps.

[0006] Low update frequency and poor timeliness: Map data relies on offline processing and release cycles. When real-world roads change (such as temporary construction, traffic control, or lane marking re-marking), maps cannot be updated in real-time or near real-time. This data lag poses serious safety hazards to autonomous driving systems.

[0007] The processing flow is complex and non-integrated: Most vision-based solutions first perform pixel-level semantic segmentation on the image to obtain a "mask" image of road elements. Subsequently, a series of complex, non-learning-based traditional post-processing algorithms (such as image morphology processing, skeleton extraction, curve fitting, etc.) are required to convert the pixel information into vectorized geometric lines. This multi-stage processing flow is prone to error accumulation.

[0008] Separation of geometry and topology generation: In existing technological frameworks, the geometric shape of a road (where the lines are) and the topological relationships (how the lines connect) are two separate tasks. Algorithms typically generate geometric lines first, and then define the topological connections through a separate rule engine or manual intervention. This separated paradigm cannot guarantee the inherent consistency between geometry and topology, and further increases the complexity of the system.

[0009] Therefore, there is an urgent need in this field for a technical solution that can automatically, efficiently, and in an integrated manner generate accurate road vector geometry and topology information using low-cost roadside cameras. Summary of the Invention

[0010] To overcome the shortcomings of the prior art, the present invention provides a method and apparatus for generating road maps based on an end-to-end visual neural network. This method is an end-to-end road geometry and topology generation technology based on multi-camera fusion, which solves the technical problems of high production cost, poor update timeliness, and separation of geometry and topology generation processes in the prior art.

[0011] The core of this invention is to construct a unified generative neural network model that integrates image features from multiple roadside cameras into a unified bird's-eye view (BEV) space, and to design a joint decoder that enables the model to simultaneously learn and output vectorized road geometry and topological connections between instances within an end-to-end framework, thereby ensuring their inherent consistency.

[0012] The technical solution provided by this invention is:

[0013] A method for generating road maps based on an end-to-end visual neural network is proposed. This method generates road geometry and topology maps end-to-end based on multi-camera fusion image features. The core of this method lies in executing a complete data processing flow. By constructing and training a single end-to-end road geometry and topology map generation model, map elements are directly output from multi-view images. Specifically, this method includes the following four execution stages:

[0014] Multi-view image synchronous acquisition and feature extraction stage: Multiple roadside cameras deployed around the intersection synchronously acquire scattered multi-view images at the same time. Each acquired frame is input into a shared-weight image encoder backbone network to extract multi-scale two-dimensional depth feature maps. The shared-weight design ensures that images from different perspectives undergo feature space transformation using a unified standard.

[0015] The view-space transformation and temporal bird's-eye view feature fusion stage involves: first, using a depth prediction network to estimate the actual physical depth distribution of each feature pixel in the 2D feature map; then, combining the camera's extrinsic parameters, projecting the 2D feature points with depth information onto a unified world 3D coordinate system; next, performing voxel pooling along the vertical direction to compress the 3D feature points into a pre-defined 2D bird's-eye view grid, forming a single-moment bird's-eye view feature; finally, inputting the single-moment bird's-eye view features from multiple consecutive frames into a temporal recurrent neural network, fusing feature information from historical moments, and outputting a global bird's-eye view feature base map containing spatiotemporal context. This base map is the sole feature source upon which subsequent map generation depends.

[0016] The geometric and topological joint decoding stage based on map element probes: This stage is completed in a parallel decoding architecture. First, a fixed number of learnable map element probes are initialized, each probe associated with a dynamic anchor point on the bird's-eye view plane; these probes are then coupled with global bird's-eye view feature base... Figure 1 The probes are fed into a multi-layered stacked attention mechanism decoder. Inside the decoder, the probes adaptively extract local relevant features from the bird's-eye view feature base map through a cross-attention mechanism, and progressively correct the positions of their dynamic anchor points, ultimately outputting final element feature embeddings rich in map element semantics and spatial location. Subsequently, the final element feature embeddings are simultaneously input into two parallel processing branches: in the geometry decoding branch, a multilayer perceptron is used to predict the road element category corresponding to each probe and generate the corresponding vectorized geometric coordinate point sequence; in the topology decoding branch, the feature embeddings of each probe are used as nodes to construct an interaction graph, extract the relationship features of paired elements, and use a multilayer perceptron to predict the connectivity confidence between each pair of elements, generating a topological adjacency matrix.

[0017] End-to-end joint optimization and map feature output stage: During model training, a bipartite graph matching algorithm is used to perform a one-to-one matching between the model's predicted road element set and the ground truth labels. After matching, classification loss, geometric regression loss (measuring spatial coordinate deviation), and topological cross-entropy loss (measuring the accuracy of connectivity judgment) are calculated. The weighted sum of these three losses is used as the total loss, and the parameters of the entire end-to-end road geometry and topology map generation model are updated through the backpropagation algorithm.

[0018] By following the steps above, it is possible to generate a high-precision intersection map that contains both accurate vectorized geometry and complete topological relationships directly from multiple roadside camera video streams based on an end-to-end visual neural network.

[0019] In specific implementation, the present invention realizes a device for generating road maps based on an end-to-end visual neural network;

[0020] This device is a dedicated neural network device for implementing the above methods. It can be deployed on roadside edge computing units (such as NVIDIA Jetson AGX Orin) or cloud servers to provide the system with scalable high-precision map infrastructure. The device consists of five core parts: raw data input layer, perception and fusion front end, view space transformation and temporal BEV fusion module, geometry and topology joint decoding back end, and output and application layer.

[0021] The core of this invention lies in the "view space transformation and temporal bird's-eye view fusion module" and the "geometric and topological joint decoding backend." The former solves the unavoidable dynamic traffic flow occlusion problem in roadside views through a temporal fusion mechanism; the latter eliminates the drawbacks of separate geometric and topological processing in existing technologies, ensuring the inherent logical consistency of geometric shapes and topological connections within a unified decoding space. The specific composition and functions of each module are as follows:

[0022] Core 1: View Space Transformation and Temporal Bird's-eye View Fusion Module. This includes a depth prediction head, a 3D view frustum projection unit, a bird's-eye view mesh fusion unit, a temporal feature fusion unit, and a bird's-eye view feature encoder. The depth prediction head is a small convolutional neural network that predicts discretized depth distributions for 2D feature pixels; the 3D view frustum projection unit upscales features to 3D space and transforms them to the intersection's global coordinate system; the bird's-eye view mesh fusion unit generates a single-time-step bird's-eye view feature map through voxel pooling; the temporal feature fusion unit is constructed using a Convolutional Gated Recurrent Unit (ConvGRU) network, utilizing a gating mechanism to achieve temporal fusion of historical and current features, filling in occluded areas; the bird's-eye view feature encoder optimizes the temporally fused feature map, outputting the final global bird's-eye view feature base map.

[0023] Core component two: Joint decoding backend for geometry and topology. (Combined with...) Figure 3As shown, this module represents the detailed structure of the geometric and topological joint decoding of this invention. Its data flow, from top to bottom, comprises three main layers: the BEV feature input layer, the decoder stack, and the output head. The BEV feature input layer receives the global bird's-eye view feature map output from the bird's-eye view encoder of the previous module, and also receives a fixed number of learnable map element probes initialized by the system. Each probe is responsible for locking and extracting a specific road element (such as lane lines) in subsequent processes. The decoder stack consists of a multi-layer Transformer decoder structure stacked from layer 1 to layer N. Within the first layer (layer 1), there are three main computational units: first, a query-query self-attention unit, used to allow information exchange between map element probes to avoid repeated detection of the same road element; second, a query-bird's-eye view cross-attention unit, allowing each probe to extract and focus on the most relevant spatial region features in the global bird's-eye view feature map according to its own task; and finally, a feedforward network and residual connection unit, used for nonlinear transformation of the features and gradient vanishing prevention processing. After iterative refinement layer by layer from layer 1 to layer N, the probe features are passed to the output head. Output Head: Receives the final element feature embedding from the decoder stack. This feature embedding is fed into four parallel prediction network subheads: a classification head, used to predict the category of the road element represented by the probe; a vector generation head, used to decode and output a continuous sequence of geometric coordinates of the road element; a pairwise relation feature extractor, used to combine features from multiple probes pairwise to extract interaction relationships; and an adjacency prediction head, which receives pairwise relation features and outputs a probability matrix indicating whether two elements are topologically connected.

[0024] Output and Application Layer: This layer includes a post-processing and filtering module, a vectorized geometry database, and a road topology database. The raw prediction results from the joint geometry and topology decoding backend first enter the post-processing and filtering module, where redundant probe results with category confidence scores below a threshold are removed. After filtering, the clean set of coordinate points and category labels are formatted and persistently stored in the "vectorized geometry database"; while the adjacency matrix data representing the directed traffic relationships between road elements is stored in the "road topology database." These two databases together constitute the complete storage carrier for the high-precision map. They serve as data sources directly connected to the roadside vehicle-to-everything (V2X) broadcasting system, used to distribute real-time local high-precision map data to autonomous vehicles entering intersections, demonstrating the ultimate application value of this invention.

[0025] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0026] This invention provides a method and apparatus for generating road maps based on an end-to-end visual neural network, the technical advantages of which include:

[0027] Overcoming the limitations of in-vehicle viewing angles, this invention significantly improves map generation integrity and anti-occlusion capabilities: Traditional in-vehicle camera solutions are limited by low installation height and local field of view, making them highly susceptible to dynamic occlusion by large vehicles or dense traffic flow at complex urban intersections. This invention employs a roadside multi-camera system, providing a stable and globally redundant view from a high vantage point. Combined with a spatiotemporal feature aggregation mechanism, it can effectively see through and infer the road structure of occluded areas, significantly improving the recall rate of map elements (especially small discrete elements such as road arrows and speed bumps) and the continuity of vector line segments. End-to-end joint optimization of geometry and topology achieves a high degree of consistency in internal logic: Existing high-precision map generation processes mostly treat geometric shape extraction and topological inference as two isolated post-processing stages. This cascaded architecture easily leads to the amplification of geometric deviations in the front end during topological inference. This invention constructs a unified query inference architecture that simultaneously decodes geometric shapes and connectivity relationships within a latent feature space. Topological prior knowledge, as a powerful structural constraint, can correct breaks or overlapping misalignments in geometric generation; while precise geometric location features directly guide the logical judgment of connectivity. The two reinforce each other, fundamentally eliminating the problem of logical violations between map elements.

[0028] High cost-effectiveness, achieving LiDAR accuracy with a pure vision solution: Traditional high-precision map data acquisition heavily relies on specialized surveying vehicles equipped with high-beam LiDAR and high-precision inertial navigation systems, resulting in extremely high capital investment for initial mapping and subsequent frequent updates. Experiments show that this invention only needs to reuse existing, low-cost standard roadside surveillance cameras in cities to achieve geometric spatial accuracy statistically comparable to expensive LiDAR solutions (surpassing LiDAR baselines at a 0.5m threshold). This provides a highly commercially viable technological alternative for realizing city-level, scalable vehicle-road cooperative digital infrastructure.

[0029] Better robustness under poor environmental and complex weather conditions: Relying on the time-series convolutional recurrent memory network (ConvGRU) of the roadside system, the model can still stably output compliant road grids and topological connectivity graphs under harsh physical conditions such as no lighting at night, water surface reflection during heavy rain, heavy fog, and snow cover. Compared with traditional single-view methods, which often experience a precipitous drop in performance, this system can still achieve this by using multi-view complementarity and topological logic error correction, thus ensuring the all-weather safety of the autonomous driving perception system. Attached Figure Description

[0030] Figure 1 The structural block diagram of the device for generating road maps based on an end-to-end visual neural network provided by the present invention.

[0031] Figure 2 A flowchart illustrating the model training and inference method provided by this invention.

[0032] Figure 3 This is a detailed structural block diagram of the geometry and topology joint decoding module in this invention. Detailed Implementation

[0033] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and through embodiments, but the scope of protection of the present invention is not limited to these embodiments.

[0034] This embodiment provides a method and apparatus for generating road maps based on an end-to-end visual neural network, which generates road geometry and topology maps end-to-end based on multi-camera fusion image features.

[0035] like Figure 1 As shown, the device of the present invention can be deployed on a roadside edge computing unit (such as a device with an embedded NVIDIA Jetson AGX) or a cloud server. The device of the present invention includes an input layer, a perception and fusion front end, a view space transformation and BEV fusion module, a geometry and topology joint decoding back end, and an output and application layer;

[0036] Input layer: includes camera, timestamp synchronizer, camera calibration database module and input interface;

[0037] In this embodiment, the device connects to four roadside industrial cameras deployed at an intersection in Yizhuang, Beijing, via a gigabit Ethernet interface. The camera parameters are 1080p resolution, 60Hz frame rate, and 90° wide-angle lens. The cameras are installed at the height of utility poles in four directions of the intersection, facing the center of the intersection. The timestamp synchronizer achieves millisecond-level time synchronization of image frames from the four cameras through hardware triggering. The camera calibration database pre-stores the intrinsic and extrinsic parameters of each camera, and the extrinsic parameters are unified to the Northeast Sky global coordinate system with the center of the intersection as the origin.

[0038] The perception and fusion front end includes: an image preprocessing module, a shared weight image backbone network, and a multi-scale feature pyramid network;

[0039] The synchronized image frames first enter the image preprocessing module, where wide-angle distortion correction is performed, and the images are uniformly scaled to a resolution of 512×960. Then, pixel values ​​are normalized using the mean and variance of the ImageNet dataset. The preprocessed images are then fed into a shared-weight image backbone network. In this embodiment, the backbone network uses EfficientNet-B5 pre-trained on ImageNet, paired with a Feature Pyramid Network (FPN), to generate 2D feature maps at four scales.

[0040] Viewspace transformation and BEV fusion module. This embodiment uses LSS (Lift-Splat-Shoot) to implement viewspace transformation and BEV fusion. Specifically:

[0041] The depth prediction head is a 3-layer convolutional network that predicts the depth distribution of 64 discrete depth intervals for each feature pixel, ranging from 1m to 100m. The 3D frustum projection unit combines camera intrinsic and extrinsic parameters to upscale 2D features to 3D space and transform them to the global coordinate system. The BEV mesh fusion unit defines a 200×200 BEV mesh with a physical resolution of 0.5m, covering a 100m×100m area ±50m from the center of the intersection, and generates single-time BEV feature maps through voxel pooling. The temporal feature fusion unit uses a ConvGRU network, taking 5 consecutive frames of single-time BEV feature maps as input (corresponding to a duration of 0.5s and 10FPS) to complete the spatiotemporal feature fusion. Finally, a 256-channel global BEV feature base map is output through a BEV feature encoder composed of 4 layers of residual blocks.

[0042] The joint decoding backend, namely the geometry and topology joint decoding module: Figure 3 The diagram shows the detailed structural composition of the geometry and topology joint decoding module in this invention.

[0043] This module comprises three main layers from top to bottom: an input layer, a decoder stack, and an output header. First, the input layer receives the feature map from the BEV encoder in the previous module, and simultaneously receives a set of learnable map element probes initialized by the system. In this embodiment, the number of probes is fixed at 300, representing the model's ability to simultaneously detect up to 300 road elements, including lane lines and stop lines. Subsequently, the data from the input layer enters a decoder stack consisting of N layers (N=6 in this embodiment) of Transformer decoder layers. In each decoder layer (such as...)... Figure 3Within layer 1), the probes sequentially pass through three main computational units: The first step, through a query-query self-attention mechanism, allows 300 probes to communicate and interact to understand their relationships, such as avoiding different probes repeatedly detecting the same lane line; the second step, through a query-BEV cross-attention mechanism, allows each probe to query and focus on the spatial region in the BEV feature map most relevant to its specific task; the third step, through a feedforward network (FFN) and residual connections, performs nonlinear transformations and enhancements on the extracted features. After N layers of decoding iterations, the probe outputs the final elemental feature embeddings, which are then fed into the output head. The output head comprises four parallel prediction subheads for joint decoding. The classification head, based on a multilayer perceptron (MLP), predicts 12 labels for the road element corresponding to the probe, including lane center line, road boundary, solid lane line, dashed lane line, stop line, pedestrian crossing, speed bump, directional arrow, bicycle lane, bus lane, median, and background. The vector generation head, based on a recurrent neural network or multilayer perceptron (RNN / MLP), predicts 20 ordered coordinate points for each element represented by the probe, which are then connected to form vectorized geometric lines. The pairwise relation feature extractor is responsible for concatenating and combining the feature embeddings of multiple probes after matching to extract the pairwise interaction features between elements. The adjacency prediction head, based on a multilayer perceptron (MLP), receives the pairwise relation features and outputs the connectivity confidence of the element pairs, ultimately generating a 200×200 topological adjacency matrix to represent the directed traversable connections between road elements.

[0044] The output and application layer includes post-processing and filtering modules, a vectorized geometry database, and a road topology database.

[0045] The raw prediction results (containing 300 feature elements) from the output head of the joint geometry and topology decoding module first enter the post-processing and filtering module. Here, invalid probe results with a class confidence score below a specific threshold (e.g., 0.3) are filtered out, and non-maximum suppression (NMS) is applied to eliminate highly overlapping prediction results. The cleaned data then flows to two interconnected databases for persistent storage, forming an application loop. Specifically, the formatted coordinate point set and corresponding class labels are received and stored in the vectorized geometry database to construct the precise spatial geometry layer of the high-precision map; the adjacency matrix data representing the connectivity of road elements is received and stored in the road topology database to construct the logical connectivity layer of the high-precision map. The interconnection of these two databases constitutes a complete structured local high-precision map data source. In practical applications, the data stored in these two databases can directly interface with the roadside vehicle-to-everything (V2X) communication system, wirelessly broadcasting intersection map elements and topological relationships to autonomous vehicles entering the intersection in real time, thereby empowering autonomous vehicles' path planning and safety decisions.

[0046] Step 1: Data Acquisition and Feature Fusion

[0047] The goal of this step is to intelligently fuse and stitch together the scattered and partial 2D image information collected by multiple cameras at different locations and angles into a unified, highly condensed scene feature base map viewed from directly above.

[0048] Multi-source image synchronous acquisition: First, the system synchronously triggers multiple cameras deployed around the intersection (e.g., on utility poles or high up on buildings) to capture images from their respective perspectives at exactly the same moment. This synchronization feature is crucial, as it ensures that all cameras are observing the scene at the same instant.

[0049] Viewpoint Feature Extraction: Each captured image frame (e.g., images from the east, south, west, and north directions) is fed into a shared image coding sub-network. This sub-network acts like a universal "eye," analyzing each image and converting it from raw pixel information into depth features that the machine can understand. "Shared" means that all images are analyzed using the same set of standards, ensuring that the extracted feature language is consistent.

[0050] Viewspace transformation and BEV bird's-eye view fusion: This is a crucial step in transitioning from 2D images to 3D space and then compressing them back into a 2D "map." The model first uses a pre-trained model to estimate the actual physical depth of features (such as multiple points on a lane line) in each 2D image, i.e., their distance from the camera. Combined with the pre-calibrated precise installation position and orientation of each camera (i.e., "extrinsic parameters"), the system accurately projects these 2D feature points with depth information to their coordinate positions in real-world 3D space. The system transforms all 3D feature points projected from the cameras into a unified, pre-defined world coordinate system. Then, the system takes a bird's-eye view, flattening all feature points in 3D space onto a 2D grid plane. During this process, feature information from different cameras regarding the same object (e.g., the same stop line or lane line) naturally falls into the same or similar positions on this 2D grid. The system intelligently fuses this overlapping information (e.g., by averaging or taking the most salient features) to form a single, global, and highly information-dense Bird's-Eye (BEV) feature representation. This BEV feature base map is the sole base map upon which subsequent mapping steps rely.

[0051] Step 2: Construct an end-to-end visual neural network to perform joint geometric and topological decoding and generate a road map;

[0052] This invention abandons the traditional multi-stage, cumbersome process of "segmenting pixels first and then fitting lines," and adopts a query vector-based set prediction paradigm. It directly decodes vectorized geometric information and topological connectivity from the BEV feature base map in parallel using a Transformer decoder, and achieves end-to-end optimization through a joint loss function. Specifically, the process includes the following:

[0053] Initialization of learnable map element query vectors: The system initializes a fixed number of learnable map element query vectors (Map Queries), each query vector is associated with a reference point on the BEV plane as a dynamic anchor point; during model training and inference, each query vector will automatically learn and lock a corresponding road element instance (such as lane center line, road boundary, stop line, pedestrian crossing, etc.) in the BEV feature base map.

[0054] Generation of vectorized geometric information: Initialized query vector and global BEV feature base Figure 1 The query vectors are fed into stacked Transformer decoder layers. Through a multi-scale deformable cross-attention mechanism, each query vector adaptively focuses on relevant regions in the BEV feature map. A coarse-to-fine layered refinement strategy is adopted, iteratively updating the features and reference point positions of the query vectors at each layer of the decoder to gradually fit the true geometric shape of the road elements. After multi-layer decoding, the final query vector is fed into two parallel geometric prediction heads: the classification head predicts the category label of the road element (such as solid lane lines, dashed lane lines, stop lines, pedestrian crossings, background, etc.) through a multi-layer perceptron (MLP); the vector generation head predicts the sequence of coordinate points relative to the final reference point through an MLP. Connecting the coordinate points in sequence constitutes the vectorized geometric shape of the road element. The geometric information of all elements together constitutes the geometric components of the high-precision map.

[0055] Joint interactive reasoning of topological relationships: The query vectors output during the geometric decoding process are synchronously fed into the topological reasoning module. This module treats each query vector as a graph node and constructs a fully connected interactive graph. Through a self-attention mechanism, global contextual interaction between query vectors is realized to understand the traffic flow logic relationship between road elements. Then, through the pairwise relation MLP prediction head, the embedded features of each pair of query vectors are input, and the connectivity confidence between two corresponding road elements is output. Finally, an adjacency matrix representing the topological connection relationship is generated, which constitutes the topological component of the high-precision map (where the nodes are road geometric elements and the edges are the passable traffic flow connection relationships between elements).

[0056] End-to-End Joint Optimization Objective Construction: To achieve end-to-end model training, a joint loss function integrating classification loss, geometric loss, and topological loss is constructed. Specifically: First, the Hungarian matching algorithm is used to achieve binary matching between predicted road elements and ground truth elements, eliminating the mismatch between the number of predicted and ground truth elements. The classification loss uses FocalLoss, which penalizes the class prediction error for successfully matched elements, addressing the class imbalance between background and effective road elements. The geometric loss uses a weighted combination of L1 loss and Chamfer Distance, penalizing the spatial deviation between the predicted geometric point sequence and the ground truth, ensuring the accuracy of vectorized geometry. The topological loss uses Binary Cross Entropy (BCE) loss, which penalizes the prediction error of topological connectivity for successfully matched element pairs, ensuring the accuracy of topological inference. The total loss is the weighted sum of the above three types of losses. Backpropagation and gradient descent algorithms are used to update all parameters of the model end-to-end, achieving joint optimization and mutual constraints between geometric and topological tasks.

[0057] By following the steps above, it is possible to generate a high-precision intersection map that contains both accurate vectorized geometry and complete topological relationships directly from multiple roadside camera video streams based on an end-to-end visual neural network.

[0058] Please see Figure 2 The method embodiments of the present invention include two stages: model building and training, and model inference.

[0059] Model training methods

[0060] For the above device to function, it must first be trained. Dataset: We use a dataset containing 5000 labeled intersection scenes. Each scene contains synchronized multi-view images and corresponding "Ground Truth" labels. The ground truth labels include: a) a set of vectorized coordinate points for each road element; b) a K×K topological adjacency matrix (K is the number of ground truth elements in the scene).

[0061] Training steps:

[0062] Forward propagation: As described in the device structure above, a batch of training images is input into the device to obtain 300 prediction results (including prediction category, prediction point set, and prediction topology matrix).

[0063] Matching: Due to the mismatch between the number of predictions and the number of true values, we use the Hungarian Algorithm to find a low-cost "one-to-one" match between predictions and true values. The cost of the matching takes into account both the accuracy of the class prediction and the accuracy of the geometric prediction.

[0064] Loss Calculation: Classification Loss For a successfully matched prediction, Focal Loss (a variant of cross-entropy loss) is used to compute the loss between it and the ground truth class; geometric loss. For a successful match prediction, a combination of L1 loss and Chamfer Distance loss is used to calculate the difference between the predicted point set and the ground truth point set; topological loss. For successfully matched prediction pairs The predicted connectivity confidence is calculated using binary cross-entropy loss. The loss between connections to the truth value (0 or 1). Total loss. The above losses are weighted and summed.

[0065]

[0066] In this embodiment, the weight parameter can be set to .

[0067] Backpropagation and updating:

[0068] Using the AdamW optimizer, the initial learning rate was set to 1e-4, and the weight decay was set to 1e-2. Training was performed on a 4-card NVIDIA V100 GPU with a batch size of 4, for a total of 50 epochs.

[0069] Model inference methods

[0070] Once the device is trained, it can be deployed to the roadside for real-time inference. The device receives video streams (e.g., 60 FPS) from multiple roadside cameras in real time, performs synchronization, preprocessing, feature extraction, and BEV fusion to generate a BEV feature map. The joint decoder performs a forward propagation on the BEV feature map, outputting the raw geometric and topological predictions in parallel. The post-processing module filters low-confidence results and outputs structured, high-precision map elements (e.g., in JSON format).

[0071] In this embodiment, the complete inference process deployed on the edge computing unit can achieve a real-time running speed of 60 FPS, realizing near real-time generation and updating of intersection map elements.

[0072] Those skilled in the art will understand that the above embodiments are merely exemplary, and various modifications and variations can be made without departing from the spirit and scope of the invention. For example, the backbone network can be EfficientNet; the BEV fusion module can also adopt the Transformer-based BEVFormer scheme; and the decoder can also adopt other types of generative models. All these variations fall within the protection scope of the present invention.

Claims

1. A method for generating road maps based on an end-to-end visual neural network, characterized in that, This method generates road geometry and topology maps end-to-end based on multi-camera fusion image features. It constructs and trains a single end-to-end road geometry and topology map generation model, directly outputting map elements from multi-view images. The steps include: 1) Perform simultaneous acquisition and feature extraction of multi-view images to obtain multi-scale two-dimensional depth feature maps; 2) Perform visual-spatial transformation and temporal bird's-eye view feature fusion, including: 21) First, a depth prediction network is used to estimate the actual physical depth distribution of each feature pixel in the two-dimensional depth feature map; 22) Project two-dimensional feature points with depth information onto a unified world three-dimensional coordinate system; 23) Perform voxel pooling along the vertical direction to compress the three-dimensional feature points into the two-dimensional bird's-eye view grid, forming the bird's-eye view features at a single time step; 24) Input the single-moment bird's-eye view features of multiple consecutive frames into the time series recurrent neural network, fuse the feature information of historical moments, and output a global bird's-eye view feature base map containing spatiotemporal context; 3) Perform joint geometric and topological decoding based on map element probes within a parallel decoding architecture; including: 31) First, initialize a set of learnable map element probes, each probe being associated with a dynamic anchor point on the bird's-eye view plane; 32) The probe and the global bird's-eye view feature base map are input into a multi-layer stacked attention mechanism decoder. Inside the decoder, the probe adaptively extracts local related features from the bird's-eye view feature base map through a cross-attention mechanism, and gradually corrects the position of the dynamic anchor point, outputting the final element feature embedding rich in map element semantics and spatial location. 33) Embed the final element features and simultaneously input them into two parallel processing branches: In the geometric decoding branch, a multilayer perceptron is used to predict the road element category corresponding to each probe and generate the corresponding vectorized geometric coordinate point sequence. In the topology decoding branch, the features of each probe are embedded as nodes to construct an interaction graph, the relationship features of paired elements are extracted, and the connectivity confidence between each pair of elements is predicted using a multilayer perceptron to generate a topology adjacency matrix, which is used to characterize the directed traversable connection relationship between road elements. 4) Perform end-to-end joint optimization and output map features, including: 41) When optimizing the training of the model, the set of road elements predicted by the model is matched one-to-one with the ground truth labels; 42) After matching, calculate the classification loss, the geometric regression loss which measures the spatial coordinate deviation, and the topological cross-entropy loss which measures the accuracy of connectivity judgment. 43) The three losses are weighted and summed to obtain the total loss, and the parameters of the entire end-to-end road geometry and topology map generation model are updated through the backpropagation algorithm; By following the steps above, a high-precision map containing both accurate vectorized geometry and complete topological relationships can be generated directly from multiple roadside camera video streams based on an end-to-end visual neural network.

2. The method for generating road maps based on an end-to-end visual neural network as described in claim 1, characterized in that, In step 1), multiple roadside cameras deployed around the intersection simultaneously collect scattered multi-view images at the same time; and each frame of the collected image is input into the backbone network of the image encoder with shared weights, so that the extracted multi-scale two-dimensional depth feature maps are transformed into feature space using a unified standard.

3. The method for generating road maps based on an end-to-end visual neural network as described in claim 1, characterized in that, In step 3), a query vector-based set prediction paradigm is adopted. The vectorized geometric information and topological connectivity are directly decoded in parallel from the BEV bird's-eye view feature base map through the Transformer decoder, and end-to-end optimization is achieved through the joint loss function.

4. The method for generating road maps based on an end-to-end visual neural network as described in claim 3, characterized in that, The query vector is fed into two parallel geometry prediction heads: The classification head predicts the category label of the road element using a multilayer perceptron (MLP); The vector generation head predicts a sequence of coordinate points relative to the final reference point using MLP. Connecting these coordinate points in sequence forms the vectorized geometry of the road element. The geometric information of all elements together constitutes the geometric components of a high-precision map.

5. The method for generating road maps based on an end-to-end visual neural network as described in claim 4, characterized in that, The specific process of topological relation joint interactive reasoning is as follows: During the geometric decoding process, query vectors are output, and each query vector is used as a graph node to construct a fully connected interactive graph. The global context interaction between query vectors is realized through a self-attention mechanism to extract the traffic flow logic relationship between road elements. Then, the embedded features of each pair of query vectors are input through the pairwise relation MLP prediction head, and the connectivity confidence between the two corresponding road elements is output to generate an adjacency matrix representing the topological connection relationship, which constitutes the topological component of the high-precision map. The nodes are road geometric elements, and the edges are the passable traffic flow connection relationship between the elements.

6. The method for generating road maps based on an end-to-end visual neural network as described in claim 5, characterized in that, The goal of constructing an end-to-end joint optimization training model is to build a joint loss function that integrates classification loss, geometric loss, and topological loss; specifically: First, the Hungarian matching algorithm is used to achieve binary matching between predicted road elements and ground truth elements, thus eliminating the problem of mismatch between the number of predicted and truth sets. The classification loss uses Focal Loss, which penalizes the category prediction error for elements that are successfully matched. The geometric loss uses a weighted combination of L1 loss and chamfer distance to penalize the spatial deviation between the predicted geometric point sequence and the ground truth. The topology loss uses binary cross-entropy loss, which penalizes the prediction error of the topology connection relationship for successfully matched element pairs. The total loss is a weighted sum of the three types of losses mentioned above. All parameters of the model are updated end-to-end through backpropagation and gradient descent algorithms, achieving joint optimization and mutual constraints of geometric and topological tasks.

7. An apparatus for generating road maps based on an end-to-end visual neural network using the method described in any one of claims 1 to 6, characterized in that, It includes an input layer, a perception and fusion front-end, a view space transformation and BEV bird's-eye view fusion module, a geometry and topology joint decoding back-end, and an output and application layer; among which: The view space transformation and BEV bird's-eye view fusion module adopts LSS to realize view space transformation and BEV fusion; it includes a depth prediction head, a 3D view frustum projection unit, a bird's-eye view mesh fusion unit, a temporal feature fusion unit, and a bird's-eye view feature encoder. The data flow of the geometry and topology joint decoding backend, from top to bottom, includes an input layer, a decoder stack, and an output head. The input layer receives the global bird's-eye view feature map output from the bird's-eye view encoder of the previous module, and also receives the initialized learnable map element probes. The decoder stack consists of multiple layers of Transformer decoder structures. The first layer of the decoder stack includes a query-query self-attention unit, a query-bird's-eye view cross-attention unit, and a feedforward network and residual connection unit. The output head receives the final element feature embedding from the decoder stack.

8. The apparatus for generating road maps based on an end-to-end visual neural network as described in claim 7, characterized in that, The input layer of the device includes a camera, a timestamp synchronizer, a camera calibration database module, and an input interface.

9. The apparatus for generating road maps based on an end-to-end visual neural network as described in claim 7, characterized in that, The depth prediction head of the view space transformation and temporal bird's-eye view fusion module uses a small convolutional neural network to predict the discretized depth distribution for two-dimensional feature pixels; The 3D view cone projection unit is used to uplift features to 3D space and transform them to the intersection's global coordinate system; The bird's-eye view mesh fusion unit is used to generate a single-time bird's-eye view feature map through voxel pooling; The temporal feature fusion unit is constructed using a convolutional gated recurrent unit network. By utilizing the gating mechanism, it achieves temporal fusion of historical features and current features to fill in the occluded areas. The bird's-eye view feature encoder is used to optimize the temporally fused feature map and output the final global bird's-eye view feature base map.

10. The apparatus for generating road maps based on an end-to-end visual neural network as described in claim 7, characterized in that, In the joint geometry and topology decoding backend, the final element feature embeddings received from the decoder stack are fed into four parallel prediction network subheadings: A classification head is used to predict the category of the road element represented by the probe; The vector generation head is used to decode and output a continuous sequence of geometric coordinate points of the road element; A pairwise relationship feature extractor is used to combine features from multiple probes in pairs to extract interaction relationships; The adjacency prediction head receives pairwise relation features and outputs a probability matrix indicating whether two elements are topologically connected.

Images