Trajectory-prior-based low-light road perception and topology inference method and system

Abstract

The application provides a low-light road perception and topology reasoning method and system based on trajectory prior, through an image pre-training model, low-light images are enhanced to restore image details and improve image brightness, while reducing distortion phenomena in the enhancement process. Moreover, crowd-sourced trajectory data is effectively used as prior information, combined with trajectory prior data to construct a rasterized heat map and vectorized trajectory information, to enhance the accuracy and robustness of the map construction process, and reduce the difficulty and cost of obtaining prior information. The scheme can effectively improve the ability to cope with automatic driving scenes under low-light and complex light conditions, and improve the road perception and topology reasoning accuracy.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and more specifically, to a method and system for low-light road perception and topological reasoning based on trajectory priors. Background Technology

[0002] In recent years, with the rise of artificial intelligence and automation, autonomous driving technology has become a crucial direction for national scientific and technological development. The rapid development of autonomous driving technology has also spurred demands for road perception and map building technologies. However, current traditional map building methods typically rely on offline mapping using LiDAR point clouds and image data collected by cameras. This approach suffers from inherent drawbacks such as high costs, delayed updates, and an inability to dynamically adapt to environmental changes, thus failing to adequately meet the needs of autonomous driving. Online mapping methods, on the other hand, utilize onboard sensors to achieve real-time environmental perception, enabling dynamic reflection of environmental changes and real-time map adjustments.

[0003] However, under low light, nighttime, or special lighting conditions, images captured by vehicle cameras often suffer from insufficient brightness, high noise, and loss of detail, making it difficult to accurately extract key road features such as lane lines, thereby affecting vehicle positioning, route planning, and decision-making.

[0004] Meanwhile, current online mapping technologies mostly use high-precision maps as prior information, but their construction and updating costs are high. With the popularization of crowdsourcing technology, a large amount of historical trajectory data has been collected. This data can reflect the actual trajectory of vehicle travel and road structure information. How to use this trajectory data as prior information and fuse it with real-time perception data to improve the accuracy of road perception and map construction, while reducing the cost of acquiring prior information, is a current technological development direction. Summary of the Invention

[0005] The purpose of this invention is to provide a low-light road perception and topological reasoning method and system based on trajectory priors, so as to enhance the quality of low-light images and improve the accuracy and robustness of map construction by combining trajectory prior data.

[0006] In a first aspect, the present invention provides a low-light road perception and topological reasoning method based on trajectory prior, the method comprising:

[0007] A surround view image captured by an onboard camera is obtained. The surround view image is a low-light image. An image pre-trained model is used to enhance the surround view image to obtain an enhanced image.

[0008] Obtain prior trajectory data, and perform rasterization encoding and vectorization encoding on the prior trajectory data respectively to generate rasterized heatmaps and vectorized trajectory information;

[0009] The enhanced image is encoded to obtain bird's-eye view features, and the bird's-eye view features are fused and aligned with the rasterized heatmap to obtain aligned fused features;

[0010] The vectorized trajectory information and the fused features are input into the decoder and decoded through a cross-attention mechanism. The decoded results are then transformed through a linear layer to generate road perception and topology inference results, which include the geometric position and topological relationship of lane segments.

[0011] In an optional implementation, the step of enhancing the panoramic image using an image pre-trained model to obtain an enhanced image includes:

[0012] The average brightness of the panoramic image is obtained by inputting the panoramic image into the image pre-trained model.

[0013] An edge map is obtained based on the average brightness of the panoramic image and a set first brightness threshold.

[0014] An initial latent representation is obtained based on the average brightness of the panoramic image and a set second brightness threshold;

[0015] An enhanced image is generated based on the edge map and the initial latent representation.

[0016] In an optional implementation, the step of obtaining an initial latent representation based on the all-around image according to the average brightness of the all-around image and a set second brightness threshold includes:

[0017] The average brightness of the panoramic image is compared with a set second brightness threshold, and a brightness-enhanced image is obtained based on the comparison result and the panoramic image.

[0018] The brightness-enhanced image is converted into a latent representation using a denoising diffusion implicit model, and random noise is generated.

[0019] The latent representation and the random noise are fused to generate an initial latent representation.

[0020] In an optional implementation, the step of generating the enhanced image based on the edge map and the initialized latent representation includes:

[0021] The initial latent representation is denoised, and self-attention features are extracted at each time step;

[0022] Predictive noise is generated based on the self-attention features, the edge map, and the initial latent representation;

[0023] The initial latent representation and the predicted noise are iteratively optimized at time steps to obtain the final latent representation. The final latent representation is then decoded to generate an enhanced image.

[0024] In an optional implementation, the trajectory prior data includes multiple trajectory data, each trajectory data consisting of multiple two-dimensional points;

[0025] The rasterized heatmap is generated in the following manner:

[0026] For each trajectory data in the prior trajectory data, obtain the orientation angle of each pair of adjacent two-dimensional points in the trajectory data;

[0027] Create a grid containing multiple grid cells, and calculate the number of trajectory data points and the average orientation angle passing through each grid cell;

[0028] The number of trajectory data for all grid cells is normalized, and the average orientation angle is transformed to a set range using the arctangent function to generate a rasterized heatmap containing density and orientation information.

[0029] The vectorized trajectory information is generated in the following way:

[0030] Representative trajectory data samples are determined from the multiple trajectory data using clustering algorithms or farthest point sampling methods;

[0031] The representative trajectory data sample is vectorized to obtain vectorized trajectory information.

[0032] In an optional implementation, the vehicle-mounted camera includes multiple cameras;

[0033] The step of encoding the enhanced image to obtain bird's-eye view features includes:

[0034] Feature extraction is performed on the enhanced images of each of the vehicle-mounted cameras to obtain the corresponding feature maps;

[0035] The obtained feature maps are input into an encoder containing multiple coding layers to perform bird’s-eye view feature extraction at multiple time steps on each feature map. In each coding layer, based on the set query information, the time information of the current time step is obtained from the bird’s-eye view features obtained in the previous time step through a time self-attention mechanism, and the spatial information of the current time step is obtained from multiple feature maps through a spatial cross-attention mechanism.

[0036] Bird's-eye view features are generated based on the temporal and spatial information output from the last coding layer.

[0037] In an optional implementation, the step of fusing and aligning the bird's-eye view features with the rasterized heatmap to obtain aligned fused features includes:

[0038] Based on the rasterized heatmap, the trajectory prior features are obtained, and the trajectory prior features are stitched together with the bird's-eye view features to obtain the stitched features;

[0039] The stitched features are processed using multiple convolutional layers to predict the coordinate offset of each pixel in the trajectory prior features;

[0040] Based on the coordinate offset of each pixel, the position information of the pixel is mapped to a new position;

[0041] The trajectory prior features are upsampled to align with the bird's-eye view features;

[0042] The aligned trajectory prior features and bird's-eye view features are weighted and fused according to the learned weights to obtain the fused features.

[0043] In an optional implementation, the step of inputting the vectorized trajectory information and the fused features into the decoder and performing decoding processing through a cross-attention mechanism includes:

[0044] The vectorized trajectory information is converted into high-dimensional vectorized trajectory information through a multilayer perceptron, and the high-dimensional vectorized trajectory information is then transformed linearly to generate an initial query.

[0045] By combining the initial query and the preset learnable query vector, a query embedding is generated;

[0046] The query embedding and the fused features are input into the decoder, and a cross-attention mechanism is used to combine the query embedding and the fused features to obtain the decoding result.

[0047] In an optional implementation, the step of generating road perception and topology inference results by transforming the decoded results through linear layers includes:

[0048] The decoded result is converted into coordinate information through linear transformation to obtain the geometric position of the lane segment;

[0049] The topological adjacency matrix of lane segments is obtained by calculating the connection relationships between lane segments using a graph neural network.

[0050] Secondly, the present invention provides a low-light road perception and topology reasoning system based on trajectory prior, the system comprising:

[0051] An enhancement processing module is used to obtain a surround view image captured by an in-vehicle camera. The surround view image is a low-light image. The module uses an image pre-trained model to enhance the surround view image to obtain an enhanced image.

[0052] The encoding module is used to obtain prior trajectory data, and to perform raster encoding and vector encoding on the prior trajectory data to generate raster heatmaps and vector trajectory information.

[0053] The fusion module is used to encode the enhanced image to obtain bird's-eye view features, and to fuse and align the bird's-eye view features with the rasterized heatmap to obtain aligned fused features.

[0054] The decoding module is used to input the vectorized trajectory information and the fused features into the decoder, perform decoding processing through a cross-attention mechanism, and generate road perception and topology inference results by transforming the decoded results through a linear layer. The road perception and topology inference results include the geometric position and topological relationship of lane segments.

[0055] The low-light road perception and topology reasoning method and system based on trajectory priors provided in this invention enhances low-light images through image pre-training models, restoring image details and improving image brightness to a certain extent while reducing image distortion during enhancement. Furthermore, it effectively utilizes crowdsourced trajectory data as prior information, combining it with trajectory prior data to construct rasterized heatmaps and vectorized trajectory information, enhancing the accuracy and robustness of map construction and reducing the difficulty and cost of acquiring prior information. This solution can effectively improve the ability to handle autonomous driving scenarios under low-light and complex lighting conditions, enhancing the accuracy of road perception and topology reasoning. Attached Figure Description

[0056] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments of the present invention will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0057] Figure 1 A flowchart of a low-light road perception and topological reasoning method based on trajectory prior provided in an embodiment of the present invention;

[0058] Figure 2 for Figure 1 A flowchart of the sub-steps included in S11;

[0059] Figure 3 for Figure 1A flowchart of the sub-steps included in S12;

[0060] Figure 4 for Figure 1 A flowchart of the sub-steps included in S13;

[0061] Figure 5 for Figure 1 Another flowchart of the sub-steps included in S13;

[0062] Figure 6 for Figure 1 A flowchart of the sub-steps included in S14;

[0063] Figure 7 for Figure 1 Another flowchart of the sub-steps included in S14;

[0064] Figure 8 A functional block diagram of a low-light road perception and topology reasoning system based on trajectory prior provided in an embodiment of the present invention;

[0065] Figure 9 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0066] The technical solutions of the present invention will now be described with reference to the accompanying drawings in the embodiments of the present invention.

[0067] Please see Figure 1 This document presents a flowchart of a low-light road perception and topology inference method based on trajectory prior, provided by an embodiment of the present invention. This method can be executed by a low-light road perception and topology inference system, which can be implemented in software and / or hardware and can be configured in an electronic device, such as a computer device, server, or, for example, a server in a back-end control platform. The detailed steps of this low-light road perception and topology inference method are described below.

[0068] S11, obtain the surround view image captured by the vehicle-mounted camera. The surround view image is a low-light image. Then, use an image pre-trained model to enhance the surround view image to obtain an enhanced image.

[0069] S12, obtain trajectory prior data, and perform rasterization encoding and vectorization encoding on the trajectory prior data respectively to generate rasterized heatmap and vectorized trajectory information.

[0070] S13, the enhanced image is encoded to obtain bird's-eye view features, and the bird's-eye view features are fused and aligned with the rasterized heatmap to obtain aligned fused features.

[0071] S14, the vectorized trajectory information and the fused features are input into the decoder, and the decoding is performed through the cross-attention mechanism. The decoded result is then transformed through a linear layer to generate road perception and topology inference results. The road perception and topology inference results include the geometric position and topological relationship of lane segments.

[0072] The vehicle is equipped with multiple onboard cameras, each capable of capturing images within its field of view. In this embodiment, the images captured by the multiple onboard cameras are combined to obtain a surround view image, which can be understood as images involving multiple directions of the vehicle. The low-light road perception and topology reasoning method provided in this embodiment can be used for perception and reasoning processing under low light, nighttime, or special lighting conditions. Therefore, the obtained surround view image can be a low-light image, i.e., an image under low light, nighttime, or special lighting conditions.

[0073] First, a pre-trained image model can be used to enhance poor-quality panoramic images, including improving image brightness and removing image noise, to obtain an enhanced image.

[0074] In addition, prior trajectory data can be extracted from a crowdsourced trajectory database, which includes multiple trajectory data points. The prior trajectory data can first be filtered, for example, by removing excessively short or abnormal trajectory data. Then, an average filter can be used to further filter the prior trajectory data, reducing random fluctuations and allowing the processed trajectory data to better represent the main movement trends.

[0075] Based on this, the prior trajectory data is rasterized and vectorized to obtain two forms of prior trajectory information, including rasterized heatmaps and vectorized trajectory information.

[0076] After obtaining the enhanced image of the panoramic view, in order to perceive the surrounding environment from a global perspective, the enhanced image can be encoded to obtain bird's-eye view features, which are the image features from the bird's-eye view perspective.

[0077] The rasterized heatmap obtained based on trajectory prior data is a raster map constructed from a global perspective. Therefore, the bird's-eye view features from the global perspective can be aligned and fused with the rasterized heatmap to obtain fused features after incorporating the bird's-eye view features into the rasterized heatmap.

[0078] Vectorized trajectory information obtained from prior trajectory data can provide vector information on the map. Therefore, the obtained fused features and vectorized trajectory information can be combined and cross-attention encoding can be performed on both to obtain road perception and topological inference results, including the geometric location and topological relationship of lane segments in the map. This can then be used for autonomous driving control guidance in autonomous driving scenarios.

[0079] The low-light road perception and topology reasoning method based on trajectory priors provided in this embodiment utilizes an image pre-trained model to enhance low-light images, thereby improving the feature extraction capability for map construction in low-light environments. This effectively enhances the ability to handle autonomous driving scenarios under low-light and complex lighting conditions, improving the accuracy of road perception and topology reasoning. Furthermore, extracting trajectory prior data as prior information reduces the cost of using high-precision maps as prior information, while simultaneously enhancing the accuracy and robustness of the map construction process.

[0080] The following provides a detailed explanation of the specific implementation methods for each of the above steps. Please refer to [link / reference]. Figure 2 The steps described above for enhancing a panoramic image using an image pre-trained model to obtain an enhanced image can be implemented in the following ways:

[0081] S111, Input the panoramic image into the image pre-training model to obtain the average brightness of the panoramic image.

[0082] S112, Based on the average brightness of the panoramic image and the set first brightness threshold, an edge map is obtained from the panoramic image.

[0083] S113, an initial potential representation is obtained based on the surrounding image according to the average brightness of the surrounding image and the set second brightness threshold.

[0084] S114, Generate an enhanced image based on the edge map and the initial latent representation.

[0085] In this embodiment, the panoramic image is input into the image pre-training model, and the average brightness x of the panoramic image can be calculated first. c .

[0086] In this embodiment, a first brightness threshold and a second brightness threshold are set. The first brightness threshold is greater than the second brightness threshold. For example, the first brightness threshold can be 70, and the second brightness threshold can be 40. The first brightness threshold can be used to extract edge information of the panoramic image, while the second brightness threshold can be used to determine the potential representation of the panoramic image.

[0087] The average brightness of the panoramic image can be compared with a first brightness threshold. If the average brightness of the panoramic image is greater than or equal to the first brightness threshold, the original panoramic image is output. If the average brightness is less than the first brightness threshold, the brightness of the panoramic image is increased proportionally to the first brightness threshold, as shown below:

[0088]

[0089] Where, x′ cx represents the brightness of the panoramic image after brightness enhancement. c τ represents the original brightness of the panoramic image. avg This indicates the first brightness threshold. This represents the average brightness of the panoramic image.

[0090] In addition, the average brightness of the panoramic image is compared with the second brightness threshold. The comparison method and the output image method are similar to those described above, and will not be repeated here.

[0091] In this embodiment, the output image obtained by processing the panoramic image based on the first brightness threshold is denoted as x′. c1 The output image obtained by processing the panoramic image based on the second brightness threshold is denoted as x′. c2 .

[0092] Using a fully nested edge detection method to analyze the output image x′ c1 Edge extraction is performed to obtain edge map e.

[0093] For the output image x′ c2 The initial latent representation of the surround view image is obtained by performing an inversion transformation using a denoising diffusion implicit model. Specifically, the step of obtaining the initial latent representation of the surround view image based on the average brightness of the image and a set second brightness threshold can be implemented in the following way:

[0094] The average brightness of the panoramic image is compared with a set second brightness threshold. Based on the comparison result and the panoramic image, a brightness-enhanced image is obtained. The brightness-enhanced image is converted into a latent representation through a denoising diffusion implicit model, and random noise is generated. The latent representation and random noise are fused to generate an initial latent representation.

[0095] In this embodiment, the average brightness of the panoramic image is compared with a second brightness threshold, and the resulting brightness-enhanced image is the aforementioned output image x′. c2 The brightened image is converted into a latent representation through a denoising diffusion implicit model inversion. Furthermore, generate random noise. in Let I represent a normal distribution with zero mean, unit variance, and independent and identically distributed dimensions, where I is the unit covariance matrix. Through adaptive instance normalization, the resulting latent representation... With random noise Perform fusion to generate an initial latent representation The fusion method is as follows:

[0096]

[0097] Where σ() represents the standard deviation function and μ() represents the mean function.

[0098] Based on this, an enhanced image is generated according to the edge map and the initialized latent representation. Specifically, this step can be achieved in the following way:

[0099] The initial latent representation is denoised, and self-attention features are extracted at each time step. Predictive noise is generated based on the self-attention features, edge map, and initial latent representation. The initial latent representation is iteratively optimized at each time step using the initial latent representation and predictive noise to obtain the final latent representation. The final latent representation is then decoded to generate an enhanced image.

[0100] In this embodiment, the denoising implicit diffusion model ∈ can first be... θ Fine-tuning can be performed using a low-rank adaptation approach. Specifically, low-rank matrices A and B can be added to the denoising implicit diffusion model, allowing it to be fine-tuned based on low-light image data without changing the weights of the original model. This results in a stable, fine-tuned denoising implicit diffusion model ∈′. θ .

[0101] Based on this, the initial latent representation of the image is... Using a stable denoising implicit diffusion model ∈′ θ Denoise the image and extract self-attention features A at each time step (t = T, ..., 1, 0) in this process. t The characteristics are as follows:

[0102]

[0103] Where, q t k t v t These are the embedding vectors for the query, key, and value, respectively, where d is the value of q. t k t v t The dimension. At the same time, q t k t v t The latent representation of the current time step can be utilized. It is generated through three different linear transformations, the specific formula of which is:

[0104]

[0105] Among them, W q W k W v These are the weight matrices for the query, key, and value, respectively, and are learnable linear parameters. Simultaneously, the self-attention feature A is extracted. t Attention features of the l-th layer and current potential representation A stable diffusion model ∈ with time step t and edge graph e as conditional inputs to control network expansion. θcn Generate prediction noise The formula is as follows:

[0106]

[0107] Then, the initial latent representation is utilized through a denoising diffusion implicit model. The predicted noise obtained in time step T according to the above formula Generate the latent representation for the next time step Then at time step T-1, using the latent representation and the prediction noise at the current time step Obtaining the latent representation Then, the latent representation is iterated and optimized according to the time step in this manner.

[0108] Finally, when the denoising process is complete (i.e., time step t = 0), the final latent representation is obtained. And decode it to obtain an enhanced image I with lower noise and normal brightness. e This serves as the output of the pre-trained model for low-light images.

[0109] Furthermore, in this embodiment, the obtained trajectory prior data is rasterized and vectorized to generate a rasterized heatmap and vectorized trajectory information. Please refer to [link to relevant documentation]. Figure 3 The steps described above for rasterizing and encoding the prior trajectory data to generate a raster heatmap can be implemented in the following ways:

[0110] S121, for each trajectory data in the prior trajectory data, obtain the orientation angle of each two adjacent two-dimensional points in the trajectory data.

[0111] S122, create a grid containing multiple grid cells, and calculate the number of trajectory data points and the average orientation angle passing through each grid cell.

[0112] S123, normalize the number of trajectory data for all grid cells, and transform the average orientation angle to a set range using the arctangent function to generate a rasterized heatmap containing density and orientation information.

[0113] In this embodiment, the prior trajectory data consists of multiple trajectory data, which can be represented as T = {P} (1) ,...,P (m)}, where, if each trajectory data P (i) Composed of n two-dimensional points, it can be represented as For each pair of adjacent two-dimensional points in the trajectory data, for example, point p i and p i+1 Let the direction angle between the two be θ. i,i+1 =arctan(y i+1 –y i ,x i+1 -x i ).

[0114] Next, a grid containing H×W cells can be created, where each cell represents a real-world location (Δx, Δy). Then, the number N of trajectory data passing through each cell and the average orientation angle θ are calculated, resulting in the maximum number N of trajectories across all cells. max Next, for the number of trajectory data points in all grid cells, the number N of trajectory data points is normalized using the Sigmoid function, as shown in the following formula:

[0115]

[0116] Then, the average direction angle θ is transformed to a set interval, such as the interval, using the arctangent function. The final result is a rasterized heatmap containing density and orientation information.

[0117] Furthermore, the step of vectorizing the prior trajectory data to generate vectorized trajectory information can be achieved in the following way:

[0118] Representative trajectory data samples are determined from multiple trajectory data using clustering algorithms or farthest point sampling methods; the representative trajectory data samples are then vectorized to obtain vectorized trajectory information.

[0119] Specifically, the most representative trajectory data samples are extracted using the K-Means clustering method or the farthest point sampling method.

[0120] For the K-Means clustering method, firstly, m trajectory data are selected from all trajectory data as the initial cluster centers. Next, P is calculated for each trajectory data. (i) Calculate the distance to each of the m cluster centers, assign it to the nearest cluster center, and then update the position of each cluster center. The specific formula is as follows:

[0121]

[0122] Where, μ (k) Let S represent the k-th cluster center. kThis represents all trajectory data assigned to the k-th class. The algorithm then iterates in this manner until the maximum number of iterations is reached or the change in cluster centers falls below a preset threshold, such as 0.0001. Finally, the cluster centers are designated as representative trajectories, forming the most representative sample of trajectory data.

[0123] For the farthest point sampling method, firstly, a random trajectory data is selected as the initial point and added to the trajectory set S. Then, the trajectory data with the largest minimum distance from all the trajectory data in the currently selected trajectory set S is selected and added to S. This process is repeated until a sufficient number of trajectory data is obtained as the most representative representative trajectory data sample.

[0124] The most representative trajectory data samples are vectorized to obtain vectorized trajectory information, which is represented as follows: Where m represents the number of trajectory data points selected, and n represents the feature dimension of the trajectory data.

[0125] Having obtained the enhanced image, rasterized heatmap, and vectorized trajectory information, the enhanced image is encoded to obtain the bird's-eye view features. (See [link to documentation]). Figure 4 This step can be achieved in the following ways:

[0126] S131, feature extraction is performed on the enhanced images of each of the vehicle-mounted cameras to obtain the corresponding feature maps.

[0127] S132, the obtained multiple feature maps are input into an encoder containing multiple coding layers to perform bird's-eye view feature extraction at multiple time steps on each feature map. In each coding layer, based on the set query information, the time information of the current time step is obtained from the bird's-eye view features obtained in the previous time step through a time self-attention mechanism, and the spatial information of the current time step is obtained from multiple feature maps through a spatial cross-attention mechanism.

[0128] S133 generates bird's-eye view features based on the temporal and spatial information output from the last coding layer.

[0129] First, the enhanced images corresponding to each vehicle camera are processed through a backbone network to extract features, resulting in feature maps for each vehicle camera. in N represents the feature map of the i-th vehicle camera. view This is the total number of vehicle-mounted cameras.

[0130] Furthermore, through a set of predefined mesh-like learnable parameters As query information, H and W represent the spatial shape of the Bird's Eye View (BEV) plane, and C represents the feature dimension. The query information can be understood as a processing unit; that is, the processing of the entire space is divided into multiple processing units, and processing is performed on a single unit at a time, which can improve processing speed and precision.

[0131] Multiple feature maps are input into the encoder, which includes multiple coding layers, for example, six coding layers. The feature maps are processed sequentially through each coding layer. The processing at each coding layer can be understood as processing at each time step. When processing is performed at each coding layer, the bird's-eye view features obtained from the previous coding layer (i.e., the previous time step) and the set query information are combined for processing.

[0132] Each encoding layer processes data from both temporal and spatial perspectives. At each encoder layer, the BEV query information Q is first used to retrieve temporal information from the bird's-eye view features obtained in the previous time step using a temporal self-attention mechanism. Then, the BEV query information Q is used to retrieve temporal information from the multi-camera features F using a spatial cross-attention mechanism. t Spatial information is queried in the middle. After passing through the feedforward network, the encoding layer outputs refined bird's-eye view features, which are then used as input for the next encoding layer. Through iterative processing of multiple encoding layers, the bird's-eye view features are gradually refined, ultimately generating a unified bird's-eye view feature for the current time step.

[0133] Based on this, please refer to Figure 5 The steps of fusing and aligning bird's-eye view features with rasterized heatmaps to obtain aligned fused features can be achieved in the following way:

[0134] S134, Based on the rasterized heatmap, the trajectory prior features are obtained, and the trajectory prior features are stitched together with the bird's-eye view features to obtain the stitched features.

[0135] S135, using multiple convolutional layers to process the spliced ​​features, predicting the coordinate offset of each pixel in the trajectory prior features.

[0136] S136, based on the coordinate offset of each pixel, map the position information of the pixel to a new position.

[0137] S137, Upsample the trajectory prior features to align them with the bird's-eye view features.

[0138] S138, the aligned trajectory prior features and bird's-eye view features are weighted and fused according to the learned weights to obtain the fused features.

[0139] In this embodiment, trajectory prior features are first obtained based on the rasterized heatmap. Trajectory prior features Bird's-eye view features The concatenation process, through multiple convolutional layers, yields concatenated features to predict the coordinate offset of each pixel within a spatial range, represented as... Since the main consideration is the planar range, C here is 2, which means the two-dimensional spatial range.

[0140] The coordinate offset includes horizontal offset and vertical offset, denoted as Δ. hw1 and Δ hw2 .

[0141] An optical flow model can be used to perform bilinear interpolation on the trajectory prior features, and the position (h, w) of each pixel can be determined based on the predicted coordinate offset (Δ). hw1 ,Δ hw2 Mapping to the new position (h′, w′), the formula is as follows:

[0142]

[0143] Where (h′, w′) represents the position (h+Δ) hw1 ,w+Δ hw2 The surrounding neighboring pixels (top left, top right, bottom left, bottom right), w p These are the weights for bilinear interpolation.

[0144] Next, the confidence fusion module is run, inputting the trajectory prior features and bird's-eye view features into the confidence fusion module. The trajectory prior features are then processed as described above. Upsampling is performed to align it with the bird's-eye view features for matching. Then, two spatial importance weight matrices α are introduced. l and β l Using 1×1 convolution kernels for bird's-eye view features and upsampled trajectory prior features Calculate the weights:

[0145]

[0146] The unnormalized weights obtained from the above equation are normalized using the Softmax function to calculate the final weights, while ensuring that α... l +β l =1, and α l ,β l ∈[0,1]:

[0147]

[0148] Finally, based on the learned weights, the two prior trajectory features and the bird's-eye view feature are weighted and fused to obtain the weighted and aligned fused feature y. l The specific formula is as follows:

[0149]

[0150] Here, ⊙ represents element-wise matrix multiplication.

[0151] Based on this, the vectorized trajectory information and fused features are input into the decoder, where they are decoded using a cross-attention mechanism. (See also...) Figure 6 This step can be achieved in the following ways:

[0152] S141, the vectorized trajectory information is converted into high-dimensional vectorized trajectory information through a multilayer perceptron, and the high-dimensional vectorized trajectory information is transformed linearly to generate an initial query.

[0153] S142, combine the initial query and the preset learnable query vector to generate a query embedding.

[0154] S143, The query embedding and the fusion feature are input into the decoder, and the decoding result is obtained by combining the query embedding and the fusion feature using a cross-attention mechanism.

[0155] Vectorize the trajectory prior information T v The trajectory information h is transformed into high-dimensional vectorized trajectory information through a multilayer perceptron (MLP). Then, the high-dimensional vectorized trajectory information h is transformed into an initial query Q through a learnable linear transformation. init The specific formula is as follows:

[0156] Q init =W q ·h+b q

[0157] Where h is the high-dimensional vectorized trajectory information, W q It is a learnable weight matrix, b q It is a bias term.

[0158] Simultaneously, the coordinates of the trajectory data are used as the initial reference points for the initial query. Finally, the initial query and the pre-defined learnable query vector are combined to form a complete query embedding. Then, the Transformer decoder, based on the Transformer architecture object detection algorithm (DETR), employs a cross-attention mechanism to combine the query embedding and enhanced fusion features to obtain the decoding result.

[0159] Please see Figure 7The step of converting the decoding results into road perception and topology inference results through linear layer transformation can be achieved in the following way:

[0160] S144 uses linear transformation to convert the decoded result into coordinate information, thus obtaining the geometric position of the lane segment.

[0161] S145, calculate the connection relationship between lane segments using a graph neural network to obtain the topological adjacency matrix of the lane segments.

[0162] In this embodiment, the decoding result from the Transformer decoder is converted into coordinate information through linear transformation to obtain the geometric position of the lane segment. Then, the connection relationship between the lane segments is calculated through a graph neural network to obtain the lane topology adjacency matrix.

[0163] The topological adjacency matrix is ​​normalized using the Sigmoid function so that the probability values ​​of the connection relationships are between 0 and 1, thereby obtaining the topological inference of the lane segment, and finally outputting the final road perception and topological inference results.

[0164] The low-light road perception and topology reasoning method based on trajectory priors provided in this embodiment enhances low-light images using a progressive diffusion approach through an image pre-trained model. This restores image details and improves image brightness while reducing image distortion during enhancement. Furthermore, by constructing a multimodal fusion system based on trajectory prior data, lane line perception and topology reasoning capabilities are enhanced, providing an effective way to improve the accuracy of road perception and topology reasoning in low-light environments.

[0165] This solution effectively enhances the ability to handle autonomous driving scenarios under low-light and complex lighting conditions, improves the accuracy of road perception and topology reasoning, and thus effectively improves the anti-interference performance of map construction in complex situations. Simultaneously, it effectively utilizes crowdsourced trajectory data as prior information, enhancing the accuracy and robustness of the map construction process while reducing the difficulty and cost of obtaining prior information.

[0166] Based on the same inventive concept, please refer to Figure 8 This invention also provides a functional module diagram of a low-light road perception and topology reasoning system based on trajectory prior. This embodiment divides the low-light road perception and topology reasoning system into functional modules according to the above method embodiments. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one processing module. The integrated modules can be implemented in hardware or as software functional modules. It should be noted that the module division in this embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0167] For example, when dividing functional modules according to their respective functions, Figure 8 The low-light road perception and topology reasoning system shown is only a schematic diagram of a device. This system may include an enhancement processing module, an encoding module, a fusion module, and a decoding module. The functions of each module of this low-light road perception and topology reasoning system will be described in detail below.

[0168] An enhancement processing module is used to obtain a surround view image captured by an in-vehicle camera. The surround view image is a low-light image. The module uses an image pre-trained model to enhance the surround view image to obtain an enhanced image.

[0169] The encoding module is used to obtain prior trajectory data, and to perform raster encoding and vector encoding on the prior trajectory data to generate raster heatmaps and vector trajectory information.

[0170] The fusion module is used to encode the enhanced image to obtain bird's-eye view features, and to fuse and align the bird's-eye view features with the rasterized heatmap to obtain aligned fused features.

[0171] The decoding module is used to input the vectorized trajectory information and the fused features into the decoder, perform decoding processing through a cross-attention mechanism, and generate road perception and topology inference results by transforming the decoded results through a linear layer. The road perception and topology inference results include the geometric position and topological relationship of lane segments.

[0172] The low-light road perception and topology reasoning system provided in this embodiment can be used to execute the low-light road perception and topology reasoning method under any of the above embodiments. For details not covered in this embodiment, please refer to the corresponding descriptions in the above embodiments. This embodiment will not elaborate further here.

[0173] Please see Figure 9 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. The electronic device can be a computer device, server, or similar component in an autonomous driving control platform. The electronic device includes a memory, a processor, and a communication module. The memory, processor, and communication module are electrically connected directly or indirectly to each other to achieve data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses or signal lines.

[0174] The memory is used to store computer programs or data. Memory can be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), etc.

[0175] The processor is used to read / write data or programs stored in the memory and execute the low-light road perception and topology reasoning method provided in any embodiment of the present invention.

[0176] The communication module is used to establish communication connections between electronic devices and other communication terminals via a network, and to send and receive data via the network.

[0177] It should be understood that, Figure 9 The structure shown is only a schematic diagram of an electronic device; the electronic device may also include components that are larger than those shown. Figure 9 The more or fewer components shown, or having the same Figure 9 The different configurations shown.

[0178] Furthermore, embodiments of the present invention also provide a computer-readable storage medium storing machine-executable instructions, which, when executed, implement the low-light road perception and topology reasoning method provided in the above embodiments.

[0179] Specifically, the computer-readable storage medium can be a general-purpose storage medium, such as a removable disk or hard disk. When the computer program on the computer-readable storage medium is executed, it can perform the aforementioned low-light road perception and topology reasoning method. The processes involved in the execution of the executable instructions on the computer-readable storage medium can be referred to the relevant descriptions in the above method embodiments, and will not be detailed here.

[0180] In the embodiments provided by this invention, it should be understood that the disclosed apparatus and method can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0181] Furthermore, the units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0182] Furthermore, the functional modules in the various embodiments of the present invention can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0183] It should be noted that if the functionality is implemented as a software module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0184] In this document, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, without necessarily requiring or implying any such actual relationship or order between these entities or operations.

[0185] The above description is merely an embodiment of the present invention and is not intended to limit the scope of protection of the present invention. For those skilled in the art, the present invention can have various modifications and variations. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A low-light road perception and topological reasoning method based on trajectory priors, characterized in that, The method includes: A surround view image captured by an onboard camera is obtained. The surround view image is a low-light image. An image pre-trained model is used to enhance the surround view image to obtain an enhanced image. Obtain prior trajectory data, and perform rasterization encoding and vectorization encoding on the prior trajectory data respectively to generate rasterized heatmaps and vectorized trajectory information; The enhanced image is encoded to obtain bird's-eye view features, and the bird's-eye view features are fused and aligned with the rasterized heatmap to obtain aligned fused features; The vectorized trajectory information and the fused features are input into the decoder and decoded through a cross-attention mechanism. The decoded results are then transformed through a linear layer to generate road perception and topology inference results, which include the geometric position and topological relationship of lane segments. The step of enhancing the panoramic image using an image pre-trained model to obtain an enhanced image includes: The average brightness of the panoramic image is obtained by inputting the panoramic image into the image pre-trained model. An edge map is obtained based on the average brightness of the panoramic image and a set first brightness threshold. An initial latent representation is obtained based on the average brightness of the panoramic image and a set second brightness threshold; An enhanced image is generated based on the edge map and the initial latent representation.

2. The low-light road perception and topological reasoning method based on trajectory prior as described in claim 1, characterized in that, The step of obtaining an initialized latent representation based on the surrounding image according to the average brightness of the surrounding image and a set second brightness threshold includes: The average brightness of the panoramic image is compared with a set second brightness threshold, and a brightness-enhanced image is obtained based on the comparison result and the panoramic image. The brightness-enhanced image is converted into a latent representation using a denoising diffusion implicit model, and random noise is generated. The latent representation and the random noise are fused to generate an initial latent representation.

3. The low-light road perception and topological reasoning method based on trajectory prior as described in claim 2, characterized in that, The step of generating an enhanced image based on the edge map and the initialized latent representation includes: The initial latent representation is denoised, and self-attention features are extracted at each time step; Predictive noise is generated based on the self-attention features, the edge map, and the initial latent representation; The initial latent representation and the predicted noise are iteratively optimized at time steps to obtain the final latent representation. The final latent representation is then decoded to generate an enhanced image.

4. The low-light road perception and topological reasoning method based on trajectory prior as described in claim 1, characterized in that, The prior trajectory data includes multiple trajectory data, each trajectory data consisting of multiple two-dimensional points; The rasterized heatmap is generated in the following manner: For each trajectory data in the prior trajectory data, obtain the orientation angle of each pair of adjacent two-dimensional points in the trajectory data; Create a grid containing multiple grid cells, and calculate the number of trajectory data points and the average orientation angle passing through each grid cell; The number of trajectory data for all grid cells is normalized, and the average orientation angle is transformed to a set range using the arctangent function to generate a rasterized heatmap containing density and orientation information. The vectorized trajectory information is generated in the following way: Representative trajectory data samples are determined from the multiple trajectory data using clustering algorithms or farthest point sampling methods; The representative trajectory data sample is vectorized to obtain vectorized trajectory information.

5. The low-light road perception and topological reasoning method based on trajectory prior as described in claim 1, characterized in that, The vehicle-mounted camera includes multiple cameras; The step of encoding the enhanced image to obtain bird's-eye view features includes: Feature extraction is performed on the enhanced images of each of the vehicle-mounted cameras to obtain the corresponding feature maps; The obtained feature maps are input into an encoder containing multiple coding layers to perform bird’s-eye view feature extraction at multiple time steps on each feature map. In each coding layer, based on the set query information, the time information of the current time step is obtained from the bird’s-eye view features obtained in the previous time step through a time self-attention mechanism, and the spatial information of the current time step is obtained from multiple feature maps through a spatial cross-attention mechanism. Bird's-eye view features are generated based on the temporal and spatial information output from the last coding layer.

6. The low-light road perception and topological reasoning method based on trajectory prior as described in claim 1, characterized in that, The step of fusing and aligning the bird's-eye view features with the rasterized heatmap to obtain aligned fused features includes: Based on the rasterized heatmap, the trajectory prior features are obtained, and the trajectory prior features are stitched together with the bird's-eye view features to obtain the stitched features; The stitched features are processed using multiple convolutional layers to predict the coordinate offset of each pixel in the trajectory prior features; Based on the coordinate offset of each pixel, the position information of the pixel is mapped to a new position; The trajectory prior features are upsampled to align with the bird's-eye view features; The aligned trajectory prior features and bird's-eye view features are weighted and fused according to the learned weights to obtain the fused features.

7. The low-light road perception and topological reasoning method based on trajectory prior as described in claim 1, characterized in that, The step of inputting the vectorized trajectory information and the fused features into the decoder and performing decoding processing through a cross-attention mechanism includes: The vectorized trajectory information is converted into high-dimensional vectorized trajectory information through a multilayer perceptron, and the high-dimensional vectorized trajectory information is then transformed linearly to generate an initial query. By combining the initial query and the preset learnable query vector, a query embedding is generated; The query embedding and the fused features are input into the decoder, and a cross-attention mechanism is used to combine the query embedding and the fused features to obtain the decoding result.

8. The low-light road perception and topological reasoning method based on trajectory prior as described in claim 1, characterized in that, The step of generating road perception and topology reasoning results by transforming the decoded results through linear layers includes: The decoded result is converted into coordinate information through linear transformation to obtain the geometric position of the lane segment; The topological adjacency matrix of lane segments is obtained by calculating the connection relationships between lane segments using a graph neural network.

9. A low-light road perception and topological reasoning system based on trajectory priors, characterized in that, The system is used to implement the low-light road perception and topology reasoning method based on trajectory prior as described in any one of claims 1-8, the system comprising: An enhancement processing module is used to obtain a surround view image captured by an in-vehicle camera. The surround view image is a low-light image. The module uses an image pre-trained model to enhance the surround view image to obtain an enhanced image. The encoding module is used to obtain prior trajectory data, and to perform raster encoding and vector encoding on the prior trajectory data to generate raster heatmaps and vector trajectory information. The fusion module is used to encode the enhanced image to obtain bird's-eye view features, and to fuse and align the bird's-eye view features with the rasterized heatmap to obtain aligned fused features. The decoding module is used to input the vectorized trajectory information and the fused features into the decoder, perform decoding processing through a cross-attention mechanism, and generate road perception and topology inference results by transforming the decoded results through a linear layer. The road perception and topology inference results include the geometric position and topological relationship of lane segments.

Images