Semantic vector map robust construction method, device, storage medium and program product

Abstract

The application relates to a semantic vector map robust construction method, device, storage medium and program product, which is applied to the automatic driving perception technical field, and the method comprises the following steps: acquiring a multi-exposure original image; performing pretreatment on the multi-exposure original image to obtain a high dynamic range image; performing BEV feature extraction and enhancement on the high dynamic range image to obtain multi-scale BEV features and enhanced BEV features; processing the enhanced BEV features to obtain a mixed query; processing the mixed query and the multi-scale BEV features through a multi-layer mask activated instance decoder to obtain an embedding query and a coordinate query; processing the enhanced BEV features through a mask feature construction unit to obtain a binary mask feature; determining a local block feature based on the coordinate query; adjusting lane point coordinates based on the local block feature, the embedding query and a multi-layer perception machine; and obtaining a semantic vectorization map. The application has the effects of improving the precision and robustness of map construction.

Description

Technical Field

[0001] This application relates to the technical field of autonomous driving perception, and in particular to a robust semantic vector map construction method, device, storage medium, and program product. Background Technology

[0002] With the rapid development of artificial intelligence and the widespread adoption of automation, autonomous driving technology has become an increasingly important part of national scientific and technological development and has received widespread attention. High-precision maps, as the core support for vehicle positioning, path planning, and environmental understanding, play a crucial role in the navigation and decision-making of autonomous vehicles. Therefore, the demand for high-precision map building capabilities is becoming more urgent.

[0003] In current mainstream map building technologies, offline mapping relies on LiDAR point clouds and road camera recognition, which has inherent drawbacks such as high cost, slow updates, and inability to dynamically adapt to environmental changes, thus failing to meet the needs of autonomous driving. Online mapping, on the other hand, is built in real time through onboard sensors, which can dynamically reflect environmental changes and adjust the map in real time.

[0004] However, in some high-light environments, such as direct sunlight causing camera overexposure or low-light conditions at night with oncoming vehicles using high beams, current online mapping methods such as MapTR and IPM have poor processing capabilities. This can lead to issues like lane line texture disappearing, causing false feature detections and posing a potential threat to the safety of autonomous driving. Furthermore, current online mapping methods are poor at handling certain local features of lane lines, such as intersections of two lane lines, areas with large lane line curvature angles, and lane lines meeting pedestrian crossings, resulting in relatively large mapping errors that affect the accuracy of decisions made by autonomous vehicles. Summary of the Invention

[0005] To improve the accuracy and robustness of map construction, this application provides a method, device, storage medium, and program product for robust construction of semantic vector maps.

[0006] Firstly, this application provides a robust method for constructing semantic vector maps, employing the following technical solution:

[0007] A robust method for constructing semantic vector maps includes:

[0008] Acquire the original image after multiple exposures;

[0009] The multi-exposure original image is preprocessed to obtain a high dynamic range image;

[0010] The high dynamic range image is subjected to BEV feature extraction and enhancement to obtain multi-scale BEV features and enhanced BEV features;

[0011] The enhanced BEV features are processed to obtain a hybrid query. The hybrid query and the multi-scale BEV features are processed by a multi-layer mask-activated instance decoder to obtain an embedded query and a coordinate query. The coordinate query includes lane point coordinates.

[0012] The enhanced BEV features are processed by a mask feature construction unit to obtain binary mask features. Local block features are determined based on the coordinate query. The lane point coordinates are adjusted based on the local block features, the embedding query, and the multilayer perceptron to obtain a semantic vectorized map.

[0013] By employing the above technical solutions, high dynamic range (HEV) images are generated through preprocessing of multi-exposure original images, effectively improving image quality and dynamic range, and providing a richer information foundation for subsequent feature extraction. BEV feature extraction and enhancement are performed on the HEV images to generate multi-scale BEV features and enhanced BEV features, achieving comprehensive extraction and optimization of image features and enhancing feature robustness. By processing the enhanced BEV features and combining them with a multi-layer mask-activated instance decoder, embedding queries and coordinate queries containing lane point coordinates are generated, enabling accurate identification and positioning of instance objects in the image. Binary mask features are generated using a mask feature construction unit, and lane point coordinates are adjusted by combining local block features, embedding queries, and a multi-layer perceptron, ultimately generating a semantic vectorized map. This significantly improves the map's accuracy and semantic expressive power. In summary, this improves the accuracy and robustness of map construction.

[0014] Optionally, the preprocessing of the multi-exposure original image to obtain a high dynamic range image includes:

[0015] The original multi-exposure image is normalized and gamma-corrected to obtain a low dynamic range image;

[0016] Each of the low dynamic range images and the corresponding multi-exposure original images are stitched together in the channel dimension using channel stitching technology to obtain the first stitched image;

[0017] The first stitched image is aligned using an optical flow alignment unit to obtain an aligned image;

[0018] The aligned images are stitched together along the channel dimension to obtain a second stitched image;

[0019] The second stitched image is passed through multiple cascaded high-frequency enhancement units to obtain initial high-frequency features of different frequencies output by the multiple high-frequency enhancement units respectively;

[0020] Multiple initial high-frequency features are fused by element-wise addition to obtain fused high-frequency features;

[0021] Shallow features of a reference frame are extracted using convolutional blocks, where the reference frame is the original multi-exposure image with the lowest exposure.

[0022] The fused high-frequency features and the shallow features are concatenated along the channel dimension and further fused using a 3×3 convolution to obtain the first fused feature;

[0023] The high dynamic range image is obtained by mapping the number of channels of the first fused feature to the dimension of the target image using a 1×1 convolution.

[0024] By adopting the above technical solutions and employing optical flow alignment technology, the blurring and misalignment problems during multi-frame input are reduced, while also reducing the loss of early frame information caused by increased network depth in traditional methods. Initial high-frequency features of different frequencies are extracted by cascaded high-frequency enhancement units and fused using an element-wise addition method, enhancing the high-frequency details of the image and improving its spatial resolution and clarity. Convolutional blocks are introduced to extract shallow features from the reference frame and further fuse them with the fused high-frequency features, effectively combining global features and local details to generate richer first fused features. A 1×1 convolution maps the first fused features to the target image dimension, resulting in a high-quality, high dynamic range image, laying a solid foundation for subsequent BEV feature extraction and semantic vectorization map construction.

[0025] Optionally, the high-frequency enhancement unit includes an average pooling layer and a max-pooling layer. The step of passing the second stitched image through multiple cascaded high-frequency enhancement units to obtain initial high-frequency features of different frequencies output by each of the multiple high-frequency enhancement units includes:

[0026] If the high-frequency enhancement unit currently being processed is the first high-frequency enhancement unit, then the second stitched image is downsampled through the average pooling layer to obtain the first low-frequency feature; if the high-frequency enhancement unit currently being processed is not the first high-frequency enhancement unit, then the initial high-frequency feature obtained from the previous high-frequency enhancement unit is downsampled through the average pooling layer to obtain the first low-frequency feature.

[0027] The first low-frequency feature is processed through preset steps to obtain the second fused feature;

[0028] The second fused feature is downsampled through the highest pooling layer to obtain the second low-frequency feature;

[0029] The second low-frequency feature is processed through the preset steps to obtain the third fused feature;

[0030] The third fusion feature is processed through multiple cascaded convolutional enhancement blocks to obtain the initial high-frequency feature;

[0031] The step of processing the first low-frequency feature through preset steps to obtain the second fused feature includes:

[0032] The first low-frequency feature is upsampled using bilinear interpolation to obtain the original feature;

[0033] Subtracting the first low-frequency feature from the original feature yields the first high-frequency feature;

[0034] The first high-frequency feature is globally enhanced using a self-attention mechanism to obtain enhanced high-frequency features;

[0035] The enhanced high-frequency features are processed through a multi-branch block to obtain the first processed data;

[0036] The first low-frequency feature is processed by three multi-branch blocks to obtain the second processed data;

[0037] The first processed data and the upsampled second processed data are concatenated along the channel dimension to obtain the concatenated features;

[0038] The spliced ​​features are fused using channel attention and convolution to obtain the second fused feature.

[0039] By adopting the above technical solution, pooling layers are used to separate high and low frequency information, and self-attention is used to enhance high frequency details, reducing the problem of detail loss in overexposure or low light in traditional methods. In the fusion process, a multi-branch fusion structure is adopted to process high and low frequency features separately, which improves the ability to process complex structures such as lane curves, thereby enhancing the image feature extraction capability under special conditions such as night and strong light.

[0040] Optionally, the step of extracting and enhancing BEV features from the high dynamic range image to obtain multi-scale BEV features and enhanced BEV features includes:

[0041] 2D features of the high dynamic range image are extracted using a shared CNN backbone;

[0042] The 2D features are extracted into the BEV representation using deformable attention to obtain the BEV features;

[0043] The BEV features are processed by a 3-layer enhanced multi-level neck network to obtain multi-scale BEV features;

[0044] The multi-scale BEV features are upsampled using bilinear interpolation and then aligned. The aligned multi-scale BEV features have the same resolution as the BEV features.

[0045] The enhanced BEV features are obtained by aggregating and aligning the multi-scale BEV features through a concatenated 3×3 convolutional layer.

[0046] By adopting the above technical solution, a three-layer enhanced multi-level neck network can dynamically focus on key regions and optimize features, while adaptively fusing multi-scale features, which can take into account both local details and global semantics during feature extraction.

[0047] Optionally, the enhanced BEV features are processed to obtain a hybrid query. The hybrid query and the multi-scale BEV features are then processed by a multi-layer mask-activated instance decoder to obtain an embedding query and a coordinate query. The coordinate query includes lane point coordinates, including:

[0048] The enhanced BEV features are subjected to basic convolution and the sigmoid function is applied to obtain a set of instance segmentation mask images composed of instance masks;

[0049] Lane queries are generated based on the instance segmentation mask image and the enhanced BEV features;

[0050] Generate point queries based on a set of predefined learnable weights;

[0051] The point query and the lane query are fused using a multilayer perceptron to obtain a hybrid query.

[0052] An instance decoder for the L layer is constructed using a multi-scale deformable DETR structure;

[0053] The embedded query and coordinate query are determined by the instance decoder and the hybrid query.

[0054] By adopting the above technical solution, the lane query is activated by instance masking instead of fixed query, so that the query includes instance-level shape prior, which improves the modeling ability of irregular boundaries; by combining multi-scale BEV features to gradually optimize coordinates, the positioning ambiguity problem caused by sparse annotation in query initialization is reduced.

[0055] Optionally, the step of processing the enhanced BEV features through a mask feature construction unit to obtain binary mask features, determining local block features based on the coordinate query, and adjusting the lane point coordinates based on the local block features, the embedding query, and the multilayer perceptron to obtain a semantic vectorized map includes:

[0056] The enhanced BEV features are subjected to basic convolution and the sigmoid function is applied to obtain a binary mask;

[0057] Construct binary mask features based on the aforementioned binary mask;

[0058] The coordinates of the lane point are determined based on the coordinate query.

[0059] Determine a square target bounding box with a preset area centered on the coordinates of the lane point;

[0060] Local block features are determined using the square target bounding box, a specific region alignment method, and a preset function.

[0061] Based on the embedded query, the local block features, and the multi-sensor head attention refresh feature query;

[0062] By regressing lane point coordinates and classification scores from the feature query using a multilayer perceptron branch, a semantic vectorized map is obtained.

[0063] By adopting the above technical solution, local blocks of mask features are extracted through coordinate query to replace traditional sparse sampling, which reduces interference from irrelevant backgrounds and allows for better focus on local details of lane lines.

[0064] Optionally, constructing binary mask features based on the binary mask includes:

[0065] The dimension of the binary mask is encrypted using a 1×1 convolution kernel;

[0066] The encrypted binary mask, the enhanced BEV feature, and a 2-channel normalized position grid are concatenated, and the concatenated result is convolved to obtain the binary mask feature.

[0067] By adopting the above technical solution, the mask is encrypted in an upgraded dimension and then fused with BEV features, highlighting the lane line area while suppressing background noise.

[0068] Secondly, this application provides a robust semantic vector map construction system, which adopts the following technical solution:

[0069] A robust semantic vector map construction system, comprising:

[0070] The image acquisition module is used to acquire multi-exposure raw images;

[0071] The preprocessing module is used to preprocess the multi-exposure original image to obtain a high dynamic range image;

[0072] The feature extraction module is used to extract and enhance BEV features from the high dynamic range image to obtain multi-scale BEV features and enhanced BEV features.

[0073] The masking module is used to process the enhanced BEV features to obtain a hybrid query. The hybrid query and the multi-scale BEV features are processed by a multi-layer mask-activated instance decoder to obtain an embedded query and a coordinate query. The coordinate query includes lane point coordinates.

[0074] The map generation module is used to process the enhanced BEV features through the mask feature construction unit to obtain binary mask features, determine local block features based on the coordinate query, and adjust the lane point coordinates based on the local block features, the embedding query, and the multilayer perceptron to obtain a semantic vectorized map.

[0075] By employing the above technical solutions, high dynamic range (HEV) images are generated through preprocessing of multi-exposure original images, effectively improving image quality and dynamic range, and providing a richer information foundation for subsequent feature extraction. BEV feature extraction and enhancement are performed on the HEV images to generate multi-scale BEV features and enhanced BEV features, achieving comprehensive extraction and optimization of image features and enhancing feature robustness. By processing the enhanced BEV features and combining them with a multi-layer mask-activated instance decoder, embedding queries and coordinate queries containing lane point coordinates are generated, enabling accurate identification and positioning of instance objects in the image. Binary mask features are generated using a mask feature construction unit, and lane point coordinates are adjusted by combining local block features, embedding queries, and a multi-layer perceptron, ultimately generating a semantic vectorized map. This significantly improves the map's accuracy and semantic expressive power. In summary, this improves the accuracy and robustness of map construction.

[0076] Thirdly, this application provides an electronic device that adopts the following technical solution:

[0077] An electronic device includes a processor coupled to a memory;

[0078] The memory stores a computer program that can be loaded by a processor and executed by the robust semantic vector map construction method described in any of the first aspects.

[0079] Fourthly, this application provides a computer-readable storage medium, which adopts the following technical solution:

[0080] A computer-readable storage medium storing a computer program capable of being loaded by a processor and executing the robust semantic vector map construction method described in any of the first aspects.

[0081] Fifthly, this application provides a computer program product, which adopts the following technical solution:

[0082] A computer program product storing a computer program capable of being loaded by a processor and executing the robust semantic vector map construction method described in any of the first aspects. Attached Figure Description

[0083] Figure 1 This is a flowchart illustrating a robust semantic vector map construction method provided in an embodiment of this application.

[0084] Figure 2 This is a structural block diagram of a robust semantic vector map construction system provided in an embodiment of this application.

[0085] Figure 3 This is a structural block diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0086] The present application will be further described in detail below with reference to the accompanying drawings.

[0087] This application provides a robust semantic vector map construction method, which can be executed by an electronic device. The electronic device can be a server or a terminal device. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The terminal device can be a smartphone, tablet computer, desktop computer, etc., but is not limited to these.

[0088] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0089] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article, unless otherwise specified, generally indicates that the preceding and following related objects have an "or" relationship.

[0090] like Figure 1 As shown, a robust semantic vector map construction method is described in the following steps (S101-S105):

[0091] Step S101: Obtain the original image after multiple exposures.

[0092] Acquire multi-exposure raw images from the vehicle's onboard camera.

[0093] Step S102: Preprocess the original multi-exposure image to obtain a high dynamic range image.

[0094] Specifically, the multi-exposure original images are preprocessed to obtain high dynamic range (HDR) images, including: normalizing and gamma-correcting the multi-exposure original images to obtain low dynamic range (LVR) images; stitching each LVR image with its corresponding multi-exposure original image along the channel dimension using channel stitching technology to obtain a first stitched image; aligning the first stitched image using an optical flow alignment unit to obtain an aligned image; stitching the aligned image along the channel dimension to obtain a second stitched image; passing the second stitched image through multiple cascaded high-frequency enhancement units to obtain initial high-frequency features of different frequencies output by each high-frequency enhancement unit; fusing the multiple initial high-frequency features using element-wise addition to obtain fused high-frequency features; extracting shallow features from a reference frame (the multi-exposure original image with the lowest exposure) using convolutional blocks; stitching the fused high-frequency features and shallow features along the channel dimension and further fusing them using 3×3 convolution to obtain a first fused feature; and mapping the number of channels of the first fused feature to the target image dimension using 1×1 convolution to obtain the HDR image.

[0095] In this embodiment, the original multi-exposure images are sorted from low to high exposure, and the original multi-exposure image with the lowest exposure is used as the reference frame. The formulas for normalizing and gamma-correcting the original multi-exposure images are as follows: ;in, For low dynamic range images, To expose the original image more, It refers to the exposure time of the i-th multi-exposure original image. This refers to the exposure time of the first multi-exposure original image, i.e., the exposure time of the reference frame. This represents the gamma correction parameter, which can be set to 1 / 2.2.

[0096] Each low dynamic range image is stitched together with its corresponding unprocessed multi-exposure original image using channel stitching technology along the channel dimension, resulting in a first stitched image that retains both original and processed information. The first stitched image can be represented as .

[0097] The optical flow alignment unit includes a convolutional block for shallow feature extraction and a pre-trained optical flow estimation network, Spynet. The steps for aligning the first stitched image using the optical flow alignment unit are as follows: First, features are extracted from the first stitched image through a shallow convolutional network. Then, the extracted features are input into the pre-trained optical flow estimation network, Spynet, to obtain the optical flow field. Finally, the optical flow field is used to deform the first stitched image to obtain the aligned image. , that is, the output of the optical flow alignment unit, where the optical flow field is used to represent the displacement vector of each pixel in the first stitched image to the first stitched image corresponding to the reference frame.

[0098] All aligned images are stitched together along the channel dimension to obtain the second stitched image. Where B is the number of aligned images, C is the number of channels, H is the height of the second stitched image, and W is the width of the second stitched image.

[0099] In this embodiment, three cascaded high-frequency enhancement units are used as an example. The second stitched image is processed sequentially through these three cascaded high-frequency enhancement units. Each high-frequency enhancement unit outputs an initial high-frequency feature, and the initial high-frequency features output by different high-frequency enhancement units correspond to different frequencies. The initial high-frequency features output by the three different high-frequency enhancement units are fused by element-wise addition to obtain the fused high-frequency features. ; Extract shallow features of the reference frame using convolutional blocks The high-frequency features and shallow features are concatenated along the channel dimension, and the concatenated features are further fused through 3×3 convolution to obtain the first fused feature. The number of channels of the first fused feature is mapped to the preset target image dimension using 1×1 convolution to obtain a high-quality noise-free and blur-free HDR image, i.e., a high dynamic range image.

[0100] Further, the high-frequency enhancement unit includes an average pooling layer and a max-pooling layer. The second stitched image is passed through multiple cascaded high-frequency enhancement units to obtain initial high-frequency features of different frequencies output by each high-frequency enhancement unit. This includes: if the currently processed high-frequency enhancement unit is the first high-frequency enhancement unit, the second stitched image is downsampled using the average pooling layer to obtain a first low-frequency feature; if the currently processed high-frequency enhancement unit is not the first high-frequency enhancement unit, the initial high-frequency feature obtained from the previous high-frequency enhancement unit is downsampled using the average pooling layer to obtain a first low-frequency feature; the first low-frequency feature is processed through preset steps to obtain a second fused feature; the second fused feature is downsampled using the max-pooling layer to obtain a second low-frequency feature; and the second low-frequency feature is processed through preset steps to obtain... The third fusion feature is processed through multiple cascaded convolutional enhancement blocks to obtain initial high-frequency features. The first low-frequency feature is processed through preset steps to obtain the second fusion feature, including: upsampling the first low-frequency feature using bilinear interpolation to obtain the original feature; subtracting the first low-frequency feature from the original feature to obtain the first high-frequency feature; globally enhancing the first high-frequency feature using a self-attention mechanism to obtain enhanced high-frequency features; processing the enhanced high-frequency feature through a multi-branch block to obtain first processed data; processing the first low-frequency feature through three multi-branch blocks to obtain second processed data; concatenating the first processed data and the upsampled second processed data along the channel dimension to obtain concatenated features; and fusing the concatenated features through channel attention and convolution to obtain the second fusion feature.

[0101] In this embodiment, the input to the first high-frequency enhancement unit is the second stitched image, while the inputs to the other two high-frequency enhancement units are the initial high-frequency features obtained from the previous high-frequency enhancement unit. The processing procedure of the first high-frequency enhancement unit is as follows: the second stitched image is downsampled through an average pooling layer to obtain the first low-frequency features. The first low-frequency feature is processed through preset steps to obtain the second fused feature. The second fused feature is then downsampled through the highest pooling layer to obtain the second low-frequency feature. The second low-frequency feature is then processed through preset steps to obtain the third fused feature. Finally, the third fused feature is processed through multiple cascaded convolutional enhancement blocks to obtain the initial high-frequency feature. It is worth noting that, except for the input and output, the intermediate processing of the highest pooling layer is the same as that of the average pooling layer.

[0102] For a single convolutional enhancement block, the channel dimension of the input third fused feature is first adjusted by a 1×1 convolution. Then, the receptive field is expanded by a 7×7 depthwise separable convolution. Finally, the features are further fused by a convolutional feedforward network with an inverse bottleneck structure. The output feature of the convolutional feedforward network with the inverse bottleneck structure is used as the output of the convolutional enhancement block. The output result is used as the input of the convolutional enhancement block connected in series with the convolutional enhancement block (when the convolutional enhancement block is not the last convolutional enhancement block) or the final output of the convolutional enhancement block (when the convolutional enhancement block is the last convolutional enhancement block): initial high-frequency features.

[0103] The specific steps for processing the first low-frequency feature to obtain the second fused feature through preset steps are as follows: upsampling the first low-frequency feature using bilinear interpolation to obtain the original feature F; subtracting the first low-frequency feature from the original feature F. Obtain the first high-frequency feature ; and through a self-attention mechanism, the first high-frequency feature Global enhancement is performed to obtain enhanced high-frequency features. Enhanced high-frequency features through a multi-branch block (MBB) The data is processed to obtain the first processed data; the first low-frequency features are processed through three multi-branch blocks. The first processed data and the upsampled second processed data are then concatenated along the channel dimension to obtain concatenated features. These concatenated features are then fused using channel attention and convolution to obtain the second fused features. The specific calculation formula is as follows: ,

[0104] in, The first data to be processed. For the second processing of data, This is the result of upsampling the data from the second processing stage. For feature concatenation, Conv is the convolution operation, and CA is the channel attention operation.

[0105] The specific steps for processing the second low-frequency feature through preset steps to obtain the third fused feature are the same as those for "processing the first low-frequency feature through preset steps to obtain the second fused feature" mentioned above, except that the input and output are different. The specific processing procedure will not be described in detail here.

[0106] The multi-branch block consists of a dual-path (high-frequency path and low-frequency path) convolutional component. The high-frequency path uses three consecutive 3×3 convolutional layers to enhance the high-frequency details of the input features, while the low-frequency path uses a single 1×1 convolutional layer to extract the low-frequency contours and main structure of the input features. The input features of the multi-branch block are processed through the high-frequency path and the low-frequency path, respectively.

[0107] Since the other two high-frequency enhancement units have the same intermediate processing as the first high-frequency enhancement unit except for input and output, only the first high-frequency enhancement unit will be used as an example for explanation here.

[0108] Step S103: Extract and enhance BEV features from the high dynamic range image to obtain multi-scale BEV features and enhanced BEV features.

[0109] Specifically, BEV features are extracted and enhanced from high dynamic range (HDR) images to obtain multi-scale BEV features and enhanced BEV features. This includes: extracting 2D features from HDR images using a shared CNN backbone; extracting the 2D features into the BEV representation using deformable attention to obtain BEV features; processing the BEV features through a 3-layer enhanced multi-level neck network to obtain multi-scale BEV features; upsampling the multi-scale BEV features using bilinear interpolation to align the multi-scale BEV features, ensuring that the aligned multi-scale BEV features have the same resolution as the original BEV features; and aggregating the aligned multi-scale BEV features through a concatenated 3×3 convolutional layer to obtain enhanced BEV features.

[0110] In this embodiment, 2D features of the high dynamic range image are extracted using a shared CNN backbone; deformable attention is then used to extract the 2D features into the BEV representation to obtain the BEV features. ,in is the size of the BEV feature, and C is the number of channels.

[0111] A unified BEV feature is constructed by fusing channel attention and spatial attention in a three-layer enhanced multi-level neck network, thus generating multi-scale BEV features. In the 3-layer augmented multi-level neck network, each layer consists of multiple residual blocks. These residual blocks utilize a hybrid approach of channel attention and spatial attention. Residual blocks within the same layer are cascaded, as are those between layers. Each layer is processed by inputting BEV features. Or the output features of the previous layer Output the updated features of this layer. The specific update formula is as follows:

[0112] ;in, It is the updated feature of the i-th layer. Indicates channel attention. This represents spatial attention.

[0113] Multi-scale BEV features were analyzed using bilinear interpolation. Upsampling is performed to align the multi-scale BEV features, resulting in aligned multi-scale BEV features. Both are related to BEV characteristics Having the same resolution, i.e., dimension alignment was performed; the aligned multi-scale BEV features were aggregated through concatenated 3×3 convolutional layers to obtain enhanced BEV features. .

[0114] Step S104: Process the enhanced BEV features to obtain a hybrid query. Process the hybrid query and multi-scale BEV features through a multi-layer mask-activated instance decoder to obtain an embedded query and a coordinate query. The coordinate query includes lane point coordinates.

[0115] Specifically, the enhanced BEV features are processed to obtain a hybrid query. A multi-layer mask-activated instance decoder then processes the hybrid query and multi-scale BEV features to obtain an embedding query and a coordinate query. The coordinate query includes lane point coordinates. The process involves: performing basic convolution on the enhanced BEV features and applying a sigmoid function to obtain a set of instance segmentation mask images composed of instance masks; generating lane queries based on the instance segmentation mask images and enhanced BEV features; generating point queries based on a set of predefined learnable weights; fusing the point queries and lane queries using a multi-layer perceptron to obtain the hybrid query; constructing an L-layer instance decoder using a multi-scale deformable DETR structure; and determining the embedding query and coordinate query using the instance decoder and the hybrid query.

[0116] In this embodiment, the enhanced BEV features are... Perform basic convolution, and process the result using the sigmoid function to obtain a set of instance segmentation mask images composed of instance masks. Where M is the number of instance queries. Image height, Image width; segmentation mask image by instance and enhanced BEV characteristics Generate lane query The specific formula is as follows:

[0117] ;

[0118] in, Here, Sigmoid is the function, and Conv is the convolution operation. To enhance BEV characteristics The transpose of .

[0119] Input a set of predefined learnable weights into a Python library to generate point queries. Where N is a fixed number of points in each map instance, and D is the dimension of the feature, point lookup is achieved through a multilayer perceptron. Lane Inquiry Merge to obtain a hybrid query .

[0120] An instance decoder of layer L is constructed using a multi-scale deformable DETR structure, and the hybrid query is input into the constructed instance decoder to obtain the embedded query. Coordinate query Specifically, this involves taking the mixed query Q as input and processing it through each layer of the instance decoder. Each layer outputs an embedding query and a coordinate query. It is worth noting that, except for the first layer which takes the mixed query as input, the input of each other layer is the output of the previous layer. At each level n, the input is a mixed query Q or the output of the previous level. and multi-scale BEV features Interactive updates thus forming embedded queries The intermediate stage of the instance decoder assigning normalized lane point locations As an anchor point, a multilayer perceptron regression branch is added. The learned offset is used to obtain the updated lane point position (coordinate lookup). The specific formula is as follows:

[0121] ,in, This is a sparse attention extraction mechanism, the specific steps of which are: Extract the region near the point, and from within that region... and Sparse sampling is performed on the surface to extract features, thereby enabling the extraction of features. Update and get ;

[0122] ,in, For the Sigmoid function, It is the inverse function of the Sigmoid function.

[0123] The embedding query and coordinate query obtained from the last layer of the instance decoder are determined as the desired embedding query and coordinate query.

[0124] Step S105: The enhanced BEV features are processed through a mask feature construction unit to obtain binary mask features. Local block features are determined based on coordinate queries. The lane point coordinates are adjusted based on local block features, embedding queries, and multilayer perceptrons to obtain a semantic vectorized map.

[0125] Specifically, the enhanced BEV features are processed through a mask feature construction unit to obtain binary mask features. Local block features are determined based on coordinate queries. The lane point coordinates are then finely adjusted based on the local block features, embedding queries, and a multilayer perceptron to obtain a semantic vectorized map. This process includes: performing basic convolution on the enhanced BEV features and applying the sigmoid function to obtain a binary mask; constructing binary mask features based on the binary mask; determining lane point coordinates based on coordinate queries; determining a square target box with a preset area centered on the lane point coordinates; determining local block features using the square target box, a specific region alignment method, and a preset function; refreshing the feature query based on embedding queries, local block features, and multi-sensor attention; and regressing the lane point coordinates and classification scores from the feature query through a multilayer perceptron branch to obtain the semantic vectorized map.

[0126] In this embodiment, the enhanced BEV features are... Perform basic convolution, and then pass the result through the sigmoid function to obtain a binary mask. ; By using a binary mask Process the data to construct binary mask features. Determine lane point coordinates from coordinate query. Determine the coordinates of the lane points. A square target bounding box with a preset area, centered on [the target area]. The preset area is pre-defined and not specifically limited here; a region is delineated around the lane point coordinates using a square target box and a specific region alignment method, and a preset function is applied within the delineated region. Extracting local block features ; Utilizing multi-sensor head attention and embedding queries and local block features Feature query Perform a refresh; via multilayer perceptron branch and From the refreshed feature query The lane point coordinates and classification scores are regressed, and then the lane point coordinates and classification scores are used together to determine the semantic vector map.

[0127] Specifically, constructing binary mask features based on a binary mask includes: encrypting the dimension of the binary mask using a 1×1 convolution kernel; concatenating the encrypted binary mask, enhanced BEV features, and a 2-channel normalized position grid; and performing a convolution operation on the concatenated result to obtain the binary mask features.

[0128] In this embodiment, binary mask features are constructed. The specific steps are as follows: A binary mask is created using a 1×1 convolution kernel. The dimension is encrypted from 2 to 32, and then the encrypted binary mask is enhanced with BEV features. A 2-channel normalized position grid Concatenate the data and perform a convolution operation on the concatenated result to obtain the binary mask features. The specific formula is as follows:

[0129] Where D represents the encryption operation, Concat represents the concatenation operation, Conv represents the convolution operation, and 2 represents the normalized position grid of the 2-channel array. The grid is obtained by concatenating the two matrices, which represent the x and y coordinates of each pixel, using two matrices respectively, and then concatenating the two matrices according to the channel dimension.

[0130] Figure 2 This is a structural block diagram of a semantic vector map robust construction system 200 provided in an embodiment of this application.

[0131] like Figure 2 As shown, the semantic vector map robust construction system 200 mainly includes:

[0132] Image acquisition module 201 is used to acquire multi-exposure raw images;

[0133] Preprocessing module 202 is used to preprocess the multi-exposure raw image to obtain a high dynamic range image;

[0134] Feature extraction module 203 is used to extract and enhance BEV features from high dynamic range images to obtain multi-scale BEV features and enhanced BEV features.

[0135] The mask processing module 204 is used to process the enhanced BEV features to obtain a hybrid query. The hybrid query and multi-scale BEV features are processed by the instance decoder activated by multi-layer mask to obtain an embedded query and a coordinate query. The coordinate query includes lane point coordinates.

[0136] The map generation module 205 is used to process the enhanced BEV features through the mask feature construction unit to obtain binary mask features, determine local block features based on coordinate query, and adjust the lane point coordinates based on local block features, embedding query and multilayer perceptron to obtain a semantic vectorized map.

[0137] As an optional implementation of this embodiment, the preprocessing module 202 is specifically used to preprocess the multi-exposure original image to obtain a high dynamic range image, including: normalizing and gamma-correcting the multi-exposure original image to obtain a low dynamic range image; stitching each low dynamic range image with the corresponding multi-exposure original image in the channel dimension using channel stitching technology to obtain a first stitched image; aligning the first stitched image using an optical flow alignment unit to obtain an aligned image; stitching the aligned image in the channel dimension to obtain a second stitched image; passing the second stitched image through multiple cascaded high-frequency enhancement units to obtain initial high-frequency features of different frequencies output by the multiple high-frequency enhancement units respectively; fusing the multiple initial high-frequency features by element-wise addition to obtain fused high-frequency features; extracting shallow features of a reference frame through convolutional blocks, the reference frame being the multi-exposure original image with the lowest exposure; stitching the fused high-frequency features and shallow features in the channel dimension, and further fusing them through 3×3 convolution to obtain a first fused feature; and mapping the number of channels of the first fused feature to the target image dimension using 1×1 convolution to obtain a high dynamic range image.

[0138] As an optional implementation of this embodiment, the high-frequency enhancement unit includes an average pooling layer and a maximum pooling layer. The preprocessing module 202 is specifically used to pass the second stitched image through multiple cascaded high-frequency enhancement units to obtain initial high-frequency features of different frequencies output by the multiple high-frequency enhancement units, including: if the high-frequency enhancement unit currently being processed is the first high-frequency enhancement unit, then the second stitched image is downsampled through the average pooling layer to obtain a first low-frequency feature; if the high-frequency enhancement unit currently being processed is not the first high-frequency enhancement unit, then the initial high-frequency feature obtained by the previous high-frequency enhancement unit is downsampled through the average pooling layer to obtain a first low-frequency feature; the first low-frequency feature is processed through preset steps to obtain a second fused feature; the second fused feature is downsampled through the maximum pooling layer to obtain a second low-frequency feature; the second low-frequency feature is processed through preset steps to obtain a second fused feature. Low-frequency features are processed to obtain a third fused feature; the third fused feature is processed through multiple cascaded convolutional enhancement blocks to obtain an initial high-frequency feature; the first low-frequency feature is processed through preset steps to obtain a second fused feature, including: upsampling the first low-frequency feature using bilinear interpolation to obtain the original feature; subtracting the first low-frequency feature from the original feature to obtain the first high-frequency feature; globally enhancing the first high-frequency feature using a self-attention mechanism to obtain an enhanced high-frequency feature; processing the enhanced high-frequency feature through a multi-branch block to obtain first processed data; processing the first low-frequency feature through three multi-branch blocks to obtain second processed data; concatenating the first processed data and the upsampled second processed data along the channel dimension to obtain a concatenated feature; and fusing the concatenated feature through channel attention and convolution to obtain the second fused feature.

[0139] As an optional implementation of this embodiment, the feature extraction module 203 is specifically used to extract and enhance BEV features from the high dynamic range image to obtain multi-scale BEV features and enhanced BEV features, including: extracting 2D features of the high dynamic range image through a shared CNN backbone; extracting the 2D features into the BEV representation using deformable attention to obtain BEV features; processing the BEV features through a 3-layer enhanced multi-level neck network to obtain multi-scale BEV features; upsampling the multi-scale BEV features using bilinear interpolation to align the multi-scale BEV features, so that the aligned multi-scale BEV features have the same resolution as the BEV features; and aggregating the aligned multi-scale BEV features through a concatenated 3×3 convolutional layer to obtain enhanced BEV features.

[0140] As an optional implementation of this embodiment, the mask processing module 204 is specifically used to process the enhanced BEV features to obtain a hybrid query. The hybrid query and multi-scale BEV features are then processed by a multi-layer mask-activated instance decoder to obtain an embedding query and a coordinate query. The coordinate query includes lane point coordinates. This process includes: performing a basic convolution on the enhanced BEV features and applying a sigmoid function to obtain a set of instance segmentation mask maps composed of instance masks; generating a lane query based on the instance segmentation mask map and the enhanced BEV features; generating a point query based on a set of predefined learnable weights; fusing the point query and lane query using a multi-layer perceptron to obtain a hybrid query; constructing an L-layer instance decoder using a multi-scale deformable DETR structure; and determining the embedding query and coordinate query using the instance decoder and the hybrid query.

[0141] As an optional implementation of this embodiment, the map generation module 205 is specifically used to process the enhanced BEV features through the mask feature construction unit to obtain binary mask features, determine local block features based on coordinate queries, and adjust the lane point coordinates based on local block features, embedding queries, and multilayer perceptrons to obtain a semantic vectorized map. This includes: performing basic convolution on the enhanced BEV features and applying the sigmoid function to obtain a binary mask; constructing binary mask features based on the binary mask; determining lane point coordinates based on coordinate queries; determining a square target box with a preset area centered on the lane point coordinates; determining local block features through the square target box, a specific region alignment method, and a preset function; refreshing the feature query based on embedding queries, local block features, and multi-sensor attention; and regressing the lane point coordinates and classification scores from the feature query through multilayer perceptron branches to obtain a semantic vectorized map.

[0142] As an optional implementation of this embodiment, the map generation module 205 is specifically used to construct binary mask features based on a binary mask, including: encrypting the dimension of the binary mask using a 1×1 convolution kernel; concatenating the encrypted binary mask, enhanced BEV features, and a 2-channel normalized position grid, and performing a convolution operation on the concatenated result to obtain the binary mask features.

[0143] In one example, a module in any of the above systems may be one or more integrated circuits configured to implement the above methods, such as one or more application-specific integrated circuits (ASICs), or one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs), or a combination of at least two of these integrated circuit forms.

[0144] For example, when modules in a system can be implemented through a processing element scheduler, the processing element can be a general-purpose processor, such as a central processing unit (CPU) or other processor capable of calling programs. Alternatively, these modules can be integrated together to form a system-on-a-chip (SOC).

[0145] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described device and module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0146] Figure 3 This is a structural block diagram of an electronic device 300 provided in an embodiment of this application.

[0147] like Figure 3 As shown, the electronic device 300 includes a processor 301 and a memory 302, and may further include one or more of an information input / output (I / O) interface 303, a communication component 304, and a communication bus 305.

[0148] The processor 301 controls the overall operation of the electronic device 300 to complete all or part of the steps of the robust semantic vector map construction method described above. The memory 302 stores various types of data to support the operation of the electronic device 300. This data may include, for example, instructions for any application or method operating on the electronic device 300, as well as application-related data. The memory 302 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as one or more of Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0149] I / O interface 303 provides an interface between processor 301 and other interface modules, such as keyboards, mice, and buttons. These buttons can be virtual or physical. Communication component 304 is used for wired or wireless communication between electronic device 300 and other devices. Wireless communication includes Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, or 4G, or a combination thereof. Therefore, the corresponding communication component 304 may include a Wi-Fi component, a Bluetooth component, and an NFC component.

[0150] The electronic device 300 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to execute the semantic vector map robust construction method given in the above embodiments.

[0151] The communication bus 305 may include a path for transmitting information between the aforementioned components. The communication bus 305 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The communication bus 305 may be divided into an address bus, a data bus, a control bus, etc.

[0152] Electronic device 300 may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital radio receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and in-vehicle terminals (such as in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers, and may also be servers.

[0153] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described robust semantic vector map construction method.

[0154] The computer-readable storage medium may include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0155] This application also provides a computer program product storing a computer program or instructions, which, when executed by a processor, implement the steps of the above-described robust semantic vector map construction method.

[0156] Computer program products can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0157] The terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0158] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the foregoing application concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions claimed in this application.

Claims

1. A semantic vector map robust construction method, characterized in that, The method comprises the following steps: obtaining a multi-exposure original image; preprocessing the multi-exposure original image to obtain a high dynamic range image; extracting and enhancing BEV features from the high dynamic range image to obtain multi-scale BEV features and enhanced BEV features; processing the enhanced BEV features to obtain a mixed query, processing the mixed query and the multi-scale BEV features through a multi-layer mask-activated instance decoder to obtain an embedding query and a coordinate query, and the coordinate query comprising lane point coordinates; processing the enhanced BEV features through a mask feature construction unit to obtain a binary mask feature, determining a local block feature based on the coordinate query, adjusting the lane point coordinates based on the local block feature, the embedding query and a multi-layer perception to obtain a semantic vectorized map.

2. The method of claim 1, wherein, The preprocessing of the multi-exposure original image to obtain a high dynamic range image comprises the following steps: normalizing and gamma correcting the multi-exposure original image to obtain a low dynamic range image; splicing each low dynamic range image with the corresponding multi-exposure original image in the channel dimension to obtain a first spliced image; aligning the first spliced image through an optical flow alignment unit to obtain an aligned image; splicing the aligned image in the channel dimension to obtain a second spliced image; passing the second spliced image through a plurality of high-frequency enhancement units connected in series to obtain a plurality of initial high-frequency features of different frequencies output by the high-frequency enhancement units; fusing a plurality of initial high-frequency features through element-by-element addition to obtain a fused high-frequency feature; extracting a shallow feature of a reference frame through a convolution block, the reference frame being the multi-exposure original image with the lowest exposure; splicing the fused high-frequency feature and the shallow feature in the channel dimension and further fusing them through 3x3 convolution to obtain a first fused feature; mapping the channel number of the first fused feature to a target image dimension using 1x1 convolution to obtain the high dynamic range image.

3. The method of claim 2, wherein, The high-frequency enhancement unit comprises an average pooling layer and a highest pooling layer, and the passing of the second spliced image through a plurality of high-frequency enhancement units connected in series to obtain a plurality of initial high-frequency features of different frequencies output by the high-frequency enhancement units comprises the following steps: if the high-frequency enhancement unit currently being processed is the first high-frequency enhancement unit, downsampling the second spliced image through the average pooling layer to obtain a first low-frequency feature; if the high-frequency enhancement unit currently being processed is not the first high-frequency enhancement unit, downsampling the initial high-frequency feature obtained by the previous high-frequency enhancement unit through the average pooling layer to obtain the first low-frequency feature; processing the first low-frequency feature through a preset step to obtain a second fused feature; downsampling the second fused feature through the highest pooling layer to obtain a second low-frequency feature; processing the second low-frequency feature through the preset step to obtain a third fused feature; The third fusion feature is processed through a plurality of convolution enhancement blocks connected in series to obtain the initial high-frequency feature; The first low-frequency feature is processed through a preset step to obtain a second fusion feature, including: The first low-frequency feature is upsampled through a bilinear interpolation method to obtain an original feature; The original feature is subtracted from the first low-frequency feature to obtain a first high-frequency feature; The first high-frequency feature is globally enhanced through a self-attention mechanism to obtain an enhanced high-frequency feature; The enhanced high-frequency feature is processed through a multi-branch block to obtain first processing data; The first low-frequency feature is processed through three multi-branch blocks to obtain second processing data; The first processing data and the second processing data upsampled are spliced in the channel dimension to obtain spliced features; The spliced features are fused through channel attention and convolution to obtain the second fusion feature.

4. The method of claim 1, wherein, The high dynamic range image is subjected to BEV feature extraction and enhancement to obtain multi-scale BEV features and enhanced BEV features, including: The 2D features of the high dynamic range image are extracted through a shared CNN backbone; The 2D features are extracted into BEV representation using deformable attention to obtain BEV features; The BEV features are processed through a 3-layer enhanced multi-level neck network to obtain multi-scale BEV features; The multi-scale BEV features are upsampled through a bilinear interpolation method, and the multi-scale BEV features are aligned. The aligned multi-scale BEV features have the same resolution as the BEV features; The aligned multi-scale BEV features are aggregated through a 3×3 convolution layer connected in series to obtain the enhanced BEV features.

5. The method of claim 1, wherein, The enhanced BEV features are processed to obtain a mixed query. The mixed query and the multi-scale BEV features are processed through a multi-layer instance decoder with mask activation to obtain an embedding query and a coordinate query. The coordinate query includes lane point coordinates, including: The enhanced BEV features are subjected to basic convolution and a sigmoid function to obtain a set of instance segmentation mask graphs composed of instance masks; A lane query is generated based on the instance segmentation mask graph and the enhanced BEV features; A point query is generated based on a set of predefined learnable weights; The point query and the lane query are fused through a multi-layer perceptron to obtain a mixed query; An L-layer instance decoder is constructed through a multi-scale deformable DETR structure; An embedding query and a coordinate query are determined through the instance decoder and the mixed query.

6. The method of claim 1, wherein, The enhanced BEV features are processed through a mask feature construction unit to obtain a binary mask feature. A local block feature is determined based on the coordinate query. The lane point coordinates are adjusted based on the local block feature, the embedding query, and a multi-layer perceptron to obtain a semantic vectorized map, including: The enhanced BEV features are subjected to basic convolution and a sigmoid function to obtain a binary mask; A binary mask feature is constructed based on the binary mask; querying the lane point coordinate based on the coordinate; determining a square target frame of a preset area centered on the lane point coordinate; determining a local block feature through the square target frame, a specific area alignment method, and a preset function; querying a multi-sensor head attention refreshing feature based on the embedding query and the local block feature; regressing lane point coordinates and classification scores from the feature query through a multi-layer perception branch to obtain a semantic vectorized map.

7. The method of claim 6, wherein, the binary mask feature is constructed based on the binary mask, including: encrypting the dimension of the binary mask through a 1x1 convolution kernel; concatenating the encrypted binary mask, the enhanced BEV feature, and a 2-channel normalized position grid, and performing convolution operation on the concatenated result to obtain the binary mask feature.

8. An electronic device, comprising: comprising a processor coupled to a memory; the processor is configured to execute a computer program stored in the memory to enable the electronic device to perform the method of any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, comprising a computer program or instructions that, when executed on a computer, cause the computer to perform the method of any one of claims 1 to 7.

10. A computer program product, characterised in that, comprising a computer program or instructions that, when executed on a processor, implement the method of any one of claims 1 to 7.

Images