Semantic vector map robust construction method and device, storage medium and program product

Abstract

The invention relates to a semantic vector map robust construction method and device, a storage medium and a program product, which are applied to the technical field of automatic driving perception, and the method comprises the following steps: obtaining a multi-exposure original image; preprocessing the multi-exposure original image to obtain a high dynamic range image; bEV feature extraction and enhancement are carried out on the high dynamic range image to obtain a multi-scale BEV feature and an enhanced BEV feature; processing the enhanced BEV features to obtain a mixed query, and processing the mixed query and the multi-scale BEV features through a multi-layer mask activated instance decoder to obtain an embedded query and a coordinate query; processing the enhanced BEV features through a mask feature construction unit to obtain binary mask features, determining local block features based on coordinate query, and adjusting lane point coordinates based on the local block features, embedded query and a multi-layer perceptron to obtain a semantic vectorization map. The method has the effect of improving the accuracy and robustness of map construction.

Description

Technical Field

[0001] The present application relates to the technical field of autonomous driving perception, and in particular to a method, device, storage medium, and program product for robustly constructing a semantic vector map. Background Art

[0002] With the rapid development of artificial intelligence and the widespread adoption of automation, autonomous driving technology has become an increasingly important part of national scientific and technological development and has garnered widespread attention. High-precision maps, as the core support for vehicle positioning, path planning, and environmental understanding, play a vital role in autonomous vehicle navigation and decision-making. Consequently, the demand for high-precision map construction capabilities has become even more urgent.

[0003] Among the current mainstream map construction technologies, offline mapping relies on lidar point clouds and road camera recognition. It has inherent defects such as high cost, delayed updates, and inability to dynamically adapt to environmental changes. It cannot well meet the needs of autonomous driving. Online mapping is achieved in real time through on-board sensors, which can dynamically reflect environmental changes and thus adjust the map in real time.

[0004] However, in some strong lighting environments, such as direct sunlight causing camera overexposure, or low-light conditions at night with oncoming vehicles using high beams, current online mapping methods such as MapTR and IPM have poor processing capabilities. This can lead to lane texture loss and false feature detection, posing a potential threat to autonomous driving safety. Furthermore, current online mapping methods have poor processing capabilities for certain local features of lane lines. For example, mapping errors are relatively large at the intersection of two lanes, at locations where lane lines curve at large angles, and at the intersection of lanes and sidewalks, hindering the ability of autonomous vehicles to make accurate decisions. Summary of the Invention

[0005] In order to improve the accuracy and robustness of map construction, the present application provides a semantic vector map robust construction method, device, storage medium and program product.

[0006] In a first aspect, the present application provides a method for robustly constructing a semantic vector map, which adopts the following technical solutions: A robust construction method for a semantic vector map, comprising: Get multi-exposure original images; Preprocessing the multi-exposure original image to obtain a high dynamic range image; Extracting and enhancing BEV features from the high dynamic range image to obtain multi-scale BEV features and enhanced BEV features; Processing the enhanced BEV features to obtain a hybrid query, processing the hybrid query and the multi-scale BEV features through a multi-layer mask-activated instance decoder to obtain an embedded query and a coordinate query, wherein the coordinate query includes lane point coordinates; The enhanced BEV features are processed by a mask feature construction unit to obtain binary mask features, local block features are determined based on the coordinate query, and the lane point coordinates are adjusted based on the local block features, the embedding query, and a multi-layer perceptron to obtain a semantic vectorized map.

[0007] By adopting the above technical solution, high dynamic range images are generated by preprocessing multi-exposure original images, which effectively improves the image quality and dynamic range, and provides a richer information basis for subsequent feature extraction; BEV features are extracted and enhanced on high dynamic range images to generate multi-scale BEV features and enhanced BEV features, thereby achieving comprehensive extraction and optimization of image features and enhancing the robustness of features; by processing the enhanced BEV features and combining them with an instance decoder activated by multi-layer masks, embedded queries and coordinate queries containing lane point coordinates are generated, thereby achieving accurate recognition and positioning of instance objects in the image; binary mask features are generated using a mask feature construction unit, and the lane point coordinates are adjusted by combining local block features, embedded queries and a multi-layer perceptron, ultimately generating a semantic vectorized map, which significantly improves the accuracy and semantic expression ability of the map. In summary, the accuracy and robustness of map construction are improved.

[0008] Optionally, preprocessing the multi-exposure original image to obtain a high dynamic range image includes: performing normalization and gamma correction processing on the multi-exposure original image to obtain a low dynamic range image; Stitching each of the low dynamic range images with the corresponding multi-exposure original image in a channel dimension using a channel stitching technology to obtain a first stitched image; Aligning the first stitched images using an optical flow alignment unit to obtain an aligned image; Stitching the aligned images in a channel dimension to obtain a second stitched image; Passing the second stitched image through a plurality of high-frequency enhancement units connected in series to obtain initial high-frequency features of different frequencies output by the plurality of high-frequency enhancement units; Fusing the multiple initial high-frequency features by element-by-element addition to obtain a fused high-frequency feature; extracting shallow features of a reference frame through a convolution block, wherein the reference frame is the multi-exposure original image with the lowest exposure; The fused high-frequency features and the shallow features are concatenated in the channel dimension, and further fused through 3×3 convolution to obtain a first fused feature; A 1×1 convolution is used to map the number of channels of the first fusion feature to the target image dimension to obtain the high dynamic range image.

[0009] By adopting the above technical solution and optical flow alignment technology, the blur and misalignment problems during multi-frame input are reduced, while the loss of early frame information caused by the increase in network depth in traditional methods is reduced; the initial high-frequency features of different frequencies are extracted by serially connected high-frequency enhancement units, and the element-by-element addition method is used for fusion, which enhances the high-frequency detail information of the image and improves the spatial resolution and clarity of the image; the convolution block is introduced to extract the shallow features of the reference frame, and further fuses them with the fused high-frequency features, effectively combining global features and local details to generate a richer first fusion feature; the first fusion feature is mapped to the target image dimension through 1×1 convolution to obtain a high-quality high dynamic range image, which lays a solid foundation for subsequent BEV feature extraction and semantic vectorization map construction.

[0010] Optionally, the high-frequency enhancement unit includes an average pooling layer and a highest pooling layer, and the step of passing the second stitched image through a plurality of high-frequency enhancement units connected in series to obtain initial high-frequency features of different frequencies output by the plurality of high-frequency enhancement units includes: If the high-frequency enhancement unit currently being processed is the first high-frequency enhancement unit, downsampling the second stitched image through the average pooling layer to obtain a first low-frequency feature; if the high-frequency enhancement unit currently being processed is not the first high-frequency enhancement unit, downsampling the initial high-frequency feature obtained by the previous high-frequency enhancement unit through the average pooling layer to obtain the first low-frequency feature; Processing the first low-frequency feature through a preset step to obtain a second fusion feature; Downsampling the second fused feature through the highest pooling layer to obtain a second low-frequency feature; Processing the second low-frequency feature through the preset steps to obtain a third fusion feature; Processing the third fused feature through a plurality of serially connected convolution enhancement blocks to obtain the initial high-frequency feature; The step of processing the first low-frequency feature through a preset step to obtain a second fusion feature includes: Upsampling the first low-frequency feature by bilinear interpolation to obtain the original feature; Subtracting the first low-frequency feature from the original feature to obtain a first high-frequency feature; Globally enhancing the first high-frequency feature through a self-attention mechanism to obtain an enhanced high-frequency feature; Processing the enhanced high-frequency features through a multi-branch block to obtain first processed data; Processing the first low-frequency feature through three multi-branch blocks to obtain second processed data; Splicing the first processed data and the second processed data obtained by upsampling in a channel dimension to obtain a splicing feature; The spliced features are fused through channel attention and convolution to obtain the second fused features.

[0011] By adopting the above technical solution, a pooling layer is used to separate high- and low-frequency information, and self-attention is used to enhance high-frequency details. This reduces the problem of detail loss in overexposure or low-light conditions in traditional methods. A multi-branch fusion structure is used in the fusion to process high- and low-frequency features separately, improving the processing capabilities of complex structures such as lane curves, thereby enhancing the ability to extract image features in special conditions such as nighttime and strong light.

[0012] Optionally, extracting and enhancing BEV features from the high dynamic range image to obtain multi-scale BEV features and enhanced BEV features includes: Extracting 2D features of the high dynamic range image through a shared CNN backbone; Extracting the 2D features into BEV representation using deformable attention to obtain BEV features; The BEV features are processed by a 3-layer enhanced multi-level neck network to obtain multi-scale BEV features; Upsampling the multi-scale BEV features by a bilinear interpolation method, and aligning the multi-scale BEV features, so that the aligned multi-scale BEV features have the same resolution as the BEV features; The enhanced BEV features are obtained by aggregating the aligned multi-scale BEV features through the series-connected 3×3 convolutional layers.

[0013] By adopting the above technical solution, a 3-layer enhanced multi-level neck network can dynamically focus on key areas and perform feature optimization, while adaptively fusing multi-scale features, which can take into account both local details and global semantics in the feature extraction process.

[0014] Optionally, the enhanced BEV features are processed to obtain a hybrid query, and the hybrid query and the multi-scale BEV features are processed by a multi-layer mask-activated instance decoder to obtain an embedded query and a coordinate query, wherein the coordinate query includes lane point coordinates, including: Performing basic convolution on the enhanced BEV features and applying a sigmoid function to obtain a set of instance segmentation mask maps consisting of instance masks; generating a lane query based on the instance segmentation mask and the enhanced BEV features; Generate point queries based on a set of predefined learnable weights; The point query and the lane query are fused by a multi-layer perceptron to obtain a hybrid query; Construct an L-layer instance decoder through a multi-scale deformable DETR structure; An embedding query and a coordinate query are determined by the instance decoder and the hybrid query.

[0015] By adopting the above technical solution, instance masks are used to activate lane queries instead of fixed queries, so that the queries contain instance-level shape priors, improving the ability to model irregular boundaries. By combining multi-scale BEV features to gradually optimize coordinates, the positioning ambiguity caused by sparse annotations in query initialization is reduced.

[0016] Optionally, the enhanced BEV feature is processed by a mask feature construction unit to obtain a binary mask feature, a local block feature is determined based on the coordinate query, and the lane point coordinates are adjusted based on the local block feature, the embedding query, and a multi-layer perceptron to obtain a semantic vectorized map, including: Performing basic convolution on the enhanced BEV feature and applying a sigmoid function to obtain a binary mask; Constructing a binary mask feature based on the binary mask; determining the lane point coordinates based on the coordinate query; Determine a square target frame with a preset area centered on the lane point coordinates; Determine local block features by using the square target frame, the specific area alignment method, and the preset function; Refreshing feature query based on the embedding query, the local block feature, and multi-sensor head attention; Lane point coordinates and classification scores are regressed from the feature query through a multi-layer perceptron branch to obtain a semantic vectorized map.

[0017] By adopting the above technical solution, local blocks of mask features are extracted through coordinate query instead of traditional sparse sampling, which reduces irrelevant background interference and can better focus on the local details of the lane lines.

[0018] Optionally, constructing a binary mask feature based on the binary mask includes: Encrypting the dimension of the binary mask by a 1×1 convolution kernel; The encrypted binary mask, the enhanced BEV feature and a 2-channel normalized position grid are cascaded, and a convolution operation is performed on the cascade result to obtain the binary mask feature.

[0019] By adopting the above technical solution, the mask is encrypted and fused with the BEV features, which not only highlights the lane line area but also suppresses background noise.

[0020] In a second aspect, the present application provides a robust semantic vector map construction system, which adopts the following technical solutions: A robust semantic vector map construction system, comprising: An image acquisition module, used to acquire multi-exposure original images; A preprocessing module, configured to preprocess the multi-exposure original image to obtain a high dynamic range image; A feature extraction module is used to extract and enhance BEV features of the high dynamic range image to obtain multi-scale BEV features and enhanced BEV features; a mask processing module configured to process the enhanced BEV features to obtain a hybrid query, and process the hybrid query and the multi-scale BEV features through a multi-layer mask-activated instance decoder to obtain an embedded query and a coordinate query, wherein the coordinate query includes lane point coordinates; A map generation module is configured to process the enhanced BEV features through a mask feature construction unit to obtain binary mask features, determine local block features based on the coordinate query, and adjust the lane point coordinates based on the local block features, the embedding query, and a multi-layer perceptron to obtain a semantic vectorized map.

[0021] By adopting the above technical solution, high dynamic range images are generated by preprocessing multi-exposure original images, which effectively improves the image quality and dynamic range, and provides a richer information basis for subsequent feature extraction; BEV features are extracted and enhanced on high dynamic range images to generate multi-scale BEV features and enhanced BEV features, thereby achieving comprehensive extraction and optimization of image features and enhancing the robustness of features; by processing the enhanced BEV features and combining them with an instance decoder activated by multi-layer masks, embedded queries and coordinate queries containing lane point coordinates are generated, thereby achieving accurate recognition and positioning of instance objects in the image; binary mask features are generated using a mask feature construction unit, and the lane point coordinates are adjusted by combining local block features, embedded queries and a multi-layer perceptron, ultimately generating a semantic vectorized map, which significantly improves the accuracy and semantic expression ability of the map. In summary, the accuracy and robustness of map construction are improved.

[0022] In a third aspect, the present application provides an electronic device, which adopts the following technical solution: An electronic device includes a processor coupled to a memory; The memory stores a computer program that can be loaded by the processor and executes the robust construction method of the semantic vector map described in any one of the first aspects.

[0023] In a fourth aspect, the present application provides a computer-readable storage medium, which adopts the following technical solution: A computer-readable storage medium stores a computer program that can be loaded by a processor and executes the robust construction method of a semantic vector map according to any one of the first aspects.

[0024] In a fifth aspect, the present application provides a computer program product that adopts the following technical solution: A computer program product stores a computer program that can be loaded by a processor and executes the robust construction method of a semantic vector map according to any one of the first aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is a flowchart of a robust construction method for a semantic vector map provided in an embodiment of the present application.

[0026] Figure 2 This is a structural block diagram of a semantic vector map robust construction system provided in an embodiment of the present application.

[0027] Figure 3 This is a structural block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] The present application is further described in detail below with reference to the accompanying drawings.

[0029] Embodiments of the present application provide a method for robustly constructing a semantic vector map. This method can be executed by an electronic device, which can be a server or a terminal device. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The terminal device can be, but is not limited to, a smartphone, a tablet computer, or a desktop computer.

[0030] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0031] In this document, the term "and / or" simply describes a relationship between related objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document, unless otherwise specified, generally indicates an "or" relationship between the related objects.

[0032] like Figure 1 As shown, a robust construction method for semantic vector maps is described. The main process of the method is described as follows (steps S101 to S105): Step S101: Acquire a multi-exposure original image.

[0033] Get multi-exposure raw images from the vehicle camera.

[0034] Step S102: pre-process the multi-exposure original image to obtain a high dynamic range image.

[0035] Specifically, the multi-exposure original image is preprocessed to obtain a high dynamic range image, including: normalizing and gamma correcting the multi-exposure original image to obtain a low dynamic range image; splicing each low dynamic range image with the corresponding multi-exposure original image in the channel dimension using the channel stitching technology to obtain a first stitched image; aligning the first stitched image through the optical flow alignment unit to obtain an aligned image; stitching the aligned images in the channel dimension to obtain a second stitched image; passing the second stitched image through multiple high-frequency enhancement units in series to obtain initial high-frequency features of different frequencies output by multiple high-frequency enhancement units; fusing the multiple initial high-frequency features by element-by-element addition to obtain a fused high-frequency feature; extracting the shallow features of the reference frame through the convolution block, the reference frame being the multi-exposure original image with the lowest exposure; splicing the fused high-frequency features with the shallow features in the channel dimension, and further fusing them through 3×3 convolution to obtain a first fused feature; using 1×1 convolution to map the number of channels of the first fused feature to the target image dimension to obtain a high dynamic range image.

[0036] In this embodiment, the multi-exposure original images are sorted from low to high according to exposure, and the multi-exposure original image with the lowest exposure is used as a reference frame. The formula for normalizing and gamma correcting the multi-exposure original images is as follows: ;in, For low dynamic range images, For the multi-exposure original image, refers to the exposure time of the i-th multi-exposure original image, is the exposure time of the first multi-exposure original image, that is, the exposure time of the reference frame, Represents the gamma correction parameter, which can be set to 1 / 2.2.

[0037] Each low dynamic range image is stitched with the corresponding multi-exposure original image before processing in the channel dimension using the channel stitching technology to obtain the first stitched image that retains the original information and the processed information , the first stitched image can be expressed as .

[0038] The optical flow alignment unit includes a convolution block for shallow feature extraction and a pre-trained optical flow estimation network Spynet. The steps of aligning the first spliced image through the optical flow alignment unit are as follows: first, the first spliced image is passed through a shallow convolutional network to extract features, then the extracted features are input into the pre-trained optical flow estimation network Spynet to obtain an optical flow field, and finally, the optical flow field is used to deform the first spliced image to obtain an aligned image. , which is the output of the optical flow alignment unit, where the optical flow field is used to represent the displacement vector of each pixel in the first stitched image moving to the first stitched image corresponding to the reference frame.

[0039] All aligned images are stitched together in the channel dimension to obtain the second stitched image , where B is the number of aligned images, C is the number of channels, H is the height of the second stitched image, and W is the width of the second stitched image.

[0040] In this embodiment, three high-frequency enhancement units connected in series are taken as an example. The second spliced image is processed by the three high-frequency enhancement units connected in series in sequence. Each high-frequency enhancement unit outputs an initial high-frequency feature. The initial high-frequency features output by different high-frequency enhancement units correspond to different frequencies. The initial high-frequency features output by the three different high-frequency enhancement units are fused by element-by-element addition to obtain a fused high-frequency feature. ; Extract shallow features of the reference frame through convolution blocks ; The fused high-frequency features and shallow features are spliced in the channel dimension, and the spliced features are further fused through 3×3 convolution to obtain the first fused feature; 1×1 convolution is used to map the number of channels of the first fused feature to the preset target image dimension to obtain a high-quality noise-free, blur-free HDR image, that is, a high dynamic range image.

[0041] Furthermore, the high-frequency enhancement unit includes an average pooling layer and a highest pooling layer, and the second stitched image is passed through multiple high-frequency enhancement units connected in series to obtain initial high-frequency features of different frequencies output by the multiple high-frequency enhancement units, including: if the high-frequency enhancement unit currently being processed is the first high-frequency enhancement unit, the second stitched image is down-sampled through the average pooling layer to obtain a first low-frequency feature; if the high-frequency enhancement unit currently being processed is not the first high-frequency enhancement unit, the initial high-frequency feature obtained by the previous high-frequency enhancement unit is down-sampled through the average pooling layer to obtain a first low-frequency feature; the first low-frequency feature is processed through a preset step to obtain a second fused feature; the second fused feature is down-sampled through the highest pooling layer to obtain a second low-frequency feature; the second low-frequency feature is processed through a preset step to obtain a third fusion feature; processing the third fusion feature through multiple serial convolution enhancement blocks to obtain an initial high-frequency feature; processing the first low-frequency feature through preset steps to obtain a second fusion feature, including: upsampling the first low-frequency feature through bilinear interpolation to obtain the original feature; subtracting the first low-frequency feature from the original feature to obtain the first high-frequency feature; globally enhancing the first high-frequency feature through a self-attention mechanism to obtain an enhanced high-frequency feature; processing the enhanced high-frequency feature through a multi-branch block to obtain a first processed data; processing the first low-frequency feature through three multi-branch blocks to obtain a second processed data; splicing the first processed data and the upsampled second processed data in the channel dimension to obtain a spliced feature; fusing the spliced feature through channel attention and convolution to obtain a second fusion feature.

[0042] In this embodiment, the first high-frequency enhancement unit takes the second stitched image as input, and the other two high-frequency enhancement units take the initial high-frequency features obtained by the previous high-frequency enhancement unit as input. The processing process of the first high-frequency enhancement unit is as follows: downsampling the second stitched image through the average pooling layer to obtain the first low-frequency features. , the first low-frequency feature is processed through the preset steps to obtain the second fusion feature ; Then, the second fusion feature is downsampled through the highest pooling layer to obtain the second low-frequency feature, and the second low-frequency feature is processed through preset steps to obtain the third fusion feature; finally, the third fusion feature is processed through multiple series-connected convolution enhancement blocks to obtain the initial high-frequency feature. It is worth noting that, except for the input and output, the intermediate processing process of the highest pooling layer is consistent with the average pooling layer.

[0043] For a single convolution enhancement block, the third fusion feature of the input is first adjusted to the channel dimension through a 1×1 convolution, and then the receptive field is expanded through a 7×7 depth-separable convolution. Finally, the features are further fused through the convolution feedforward network with an inverse bottleneck structure, and the output features of the convolution feedforward network with an inverse bottleneck structure are used as the output of the convolution enhancement block. The output result is used as the input of the convolution enhancement block connected in series with the convolution enhancement block (when the convolution enhancement block is not the last convolution enhancement block) or the final output of the convolution enhancement block (when the convolution enhancement block is the last convolution enhancement block): the initial high-frequency features.

[0044] The specific steps of processing the first low-frequency feature through the preset steps to obtain the second fusion feature are: upsampling the first low-frequency feature through bilinear interpolation to obtain the original feature F; subtracting the first low-frequency feature from the original feature F Get the first high-frequency feature ; and through the self-attention mechanism to the first high-frequency feature Perform global enhancement to obtain enhanced high-frequency features ; Enhance high-frequency features through a multi-branch block MBB Processing is performed to obtain the first processed data; the first low-frequency feature is processed by three multi-branch blocks Process the first processed data and the second processed data through upsampling to obtain the second processed data; splice the first processed data and the second processed data through upsampling in the channel dimension to obtain the splicing feature; fuse the splicing feature through channel attention and convolution to obtain the second fused feature , the specific calculation formula is: , in, For the first processed data, For the second processing data, is the result of upsampling the second processed data, is the concatenation feature, Conv is the convolution operation, and CA is the channel attention operation.

[0045] The specific steps of processing the second low-frequency feature through the preset steps to obtain the third fusion feature are the same as the specific steps of "processing the first low-frequency feature through the preset steps to obtain the second fusion feature" mentioned above, with only the input and output being different. The specific processing process will not be repeated here.

[0046] The multi-branch block consists of a dual-path (high-frequency path and low-frequency path) convolution component, where the high-frequency path uses three consecutive 3×3 convolution layers to enhance the high-frequency details of the input features, and the low-frequency path uses a single 1×1 convolution layer to extract the low-frequency contour and main structure of the input features. The input features of the multi-branch block are processed by the high-frequency path and the low-frequency path respectively.

[0047] Since the intermediate processing of the other two high-frequency enhancement units is consistent with that of the first high-frequency enhancement unit except for input and output, only the first high-frequency enhancement unit is used as an example for description here.

[0048] Step S103: extract and enhance BEV features from the high dynamic range image to obtain multi-scale BEV features and enhanced BEV features.

[0049] Specifically, BEV features are extracted and enhanced on high dynamic range images to obtain multi-scale BEV features and enhanced BEV features, including: extracting 2D features of high dynamic range images through a shared CNN backbone; extracting 2D features into BEV representation using deformable attention to obtain BEV features; processing BEV features through a 3-layer enhanced multi-level neck network to obtain multi-scale BEV features; upsampling the multi-scale BEV features through bilinear interpolation and aligning the multi-scale BEV features, so that the aligned multi-scale BEV features have the same resolution as the BEV features; aggregating the aligned multi-scale BEV features through a series of 3×3 convolutional layers to obtain enhanced BEV features.

[0050] In this embodiment, the 2D features of the high dynamic range image are extracted through the shared CNN backbone; the 2D features are extracted into the BEV representation using deformable attention to obtain the BEV features. ,in is the size of the BEV feature, and C is the number of channels.

[0051] A unified BEV feature is constructed by fusing a 3-layer enhanced multi-level neck network with channel attention and spatial attention, that is, generating multi-scale BEV features. In the 3-layer enhanced multi-level neck network, each layer is composed of multiple residual blocks. The residual blocks are based on a mixture of channel attention and spatial attention. The residual blocks in the same layer are cascaded, and the layers are also cascaded. Each layer is fed with BEV features. Or the output features of the previous layer , output the updated features of this layer , the specific update formula is as follows: ;in, is the updated feature of layer i, represents channel attention, Indicates spatial attention.

[0052] Multi-scale BEV features are obtained by bilinear interpolation. Upsampling is performed to align the multi-scale BEV features. The aligned multi-scale BEV features Both with BEV characteristics The BEV features are aligned with the same resolution, that is, the dimensions are aligned. The enhanced BEV features are obtained by aggregating the aligned multi-scale BEV features through the 3×3 convolutional layers after concatenation. .

[0053] Step S104: Process the enhanced BEV features to obtain a hybrid query. Process the hybrid query and the multi-scale BEV features through a multi-layer mask-activated instance decoder to obtain an embedded query and a coordinate query. The coordinate query includes lane point coordinates.

[0054] Specifically, the enhanced BEV features are processed to obtain a hybrid query, and the hybrid query and multi-scale BEV features are processed by a multi-layer mask-activated instance decoder to obtain an embedded query and a coordinate query, where the coordinate query includes lane point coordinates. The method includes: performing basic convolution on the enhanced BEV features and applying a sigmoid function to obtain a set of instance segmentation mask maps composed of instance masks; generating a lane query based on the instance segmentation mask map and the enhanced BEV features; generating a point query based on a set of predefined learnable weights; fusing the point query with the lane query through a multi-layer perceptron to obtain a hybrid query; constructing an L-layer instance decoder through a multi-scale deformable DETR structure; and determining the embedded query and coordinate query through the instance decoder and the hybrid query.

[0055] In this embodiment, the BEV features are enhanced Perform basic convolution and process the convolution result through the sigmoid function to obtain a set of instance segmentation mask maps consisting of instance masks , where M is the number of instance queries, is the image height, is the image width; segmentation mask by instance and enhanced BEV features Generate Lane Query The specific formula is as follows: ;

[0056] in, is the Sigmoid function, Conv is the convolution operation, To enhance BEV characteristics The transpose of .

[0057] A set of predefined learnable weights is fed into the Python library to generate point queries. , where N is the fixed number of points per map instance and D is the dimension of the feature. The multi-layer perceptron makes the point query and lane query Fusion, get hybrid query .

[0058] The L-layer instance decoder is constructed through a multi-scale deformable DETR structure, and the mixed query is input into the constructed instance decoder to obtain the embedded query and coordinate query Specifically, it includes: taking the mixed query Q as input, processing it through each layer of the instance decoder, and each layer will output the embedded query and coordinate query. It is worth noting that except for the first layer which takes the mixed query as input, the input of each other layer is the output of the previous layer. , at each layer n, by inputting a mixed query Q or the output of the previous layer and multi-scale BEV features Interactive updates to form embedded queries , the intermediate stage where the instance decoder assigns normalized lane point locations As an anchor point, by adding a multi-layer perceptron regression branch The learned offsets are used to obtain updated lane point positions (coordinate query) , the specific formula is: ,in, It is a sparse attention extraction mechanism. The specific steps are as follows: Extract the area around the point and extract the and Sparse sampling is performed on Update and get ; ,in, is the Sigmoid function, is the inverse function of the Sigmoid function.

[0059] The embedding query and coordinate query obtained by the last layer of the instance decoder are determined as the required embedding query and coordinate query.

[0060] Step S105: Process the enhanced BEV features through the mask feature construction unit to obtain binary mask features, determine local block features based on coordinate query, adjust the lane point coordinates based on the local block features, embedding query and multi-layer perceptron to obtain a semantic vectorized map.

[0061] Specifically, the enhanced BEV features are processed by a mask feature construction unit to obtain binary mask features, local block features are determined based on coordinate queries, and the lane point coordinates are fine-tuned based on the local block features, embedding queries, and multi-layer perceptrons to obtain a semantic vectorized map, including: performing basic convolution on the enhanced BEV features and applying a sigmoid function to obtain a binary mask; constructing binary mask features based on the binary mask; determining the lane point coordinates based on the coordinate query; determining a square target box with a preset area centered on the lane point coordinates; determining local block features through the square target box, a specific area alignment method, and a preset function; refreshing the feature query based on the embedding query, local block features, and multi-sensor head attention; and regressing the lane point coordinates and classification scores from the feature query through the multi-layer perceptron branch to obtain a semantic vectorized map.

[0062] In this embodiment, the BEV features are enhanced Perform basic convolution, and pass the convolution result through the sigmoid function to obtain a binary mask ; By using binary mask Process and construct binary mask features ; Determine lane point coordinates from coordinate query ; Determine the lane point coordinates A square target box with a preset area as the center The preset area is pre-set and is not limited here. A square target frame and a specific area alignment method are used to divide an area around the lane point coordinates, and the preset function is applied in the divided area. Extract local block features ; Leverage multi-sensor head attention and embedding query and local patch features Feature query Refresh; branch through the multi-layer perceptron and Query from refreshed features The lane point coordinates and classification scores are regressed, and the lane point coordinates and classification scores are jointly determined as a semantic vectorized map.

[0063] Specifically, a binary mask feature is constructed based on the binary mask, including: encrypting the dimension of the binary mask through a 1×1 convolution kernel; cascading the encrypted binary mask, enhanced BEV features and a 2-channel normalized position grid, and performing a convolution operation on the cascaded result to obtain the binary mask feature.

[0064] In this embodiment, a binary mask feature is constructed The specific steps are: use a 1×1 convolution kernel to transform the binary mask The dimension is encrypted from 2 to 32, and then the encrypted binary mask and enhanced BEV features are and a 2-channel normalized position grid Perform cascade and convolution operation on the cascade result to obtain binary mask features , the specific formula is as follows: , where D is the encryption operation, Concat is the cascade concatenation operation, Conv is the convolution operation, and the normalized position grid of the 2 channels is The grid is formed by representing the x and y coordinates of each pixel through two matrices, and then concatenating the two matrices along the channel dimension.

[0065] Figure 2 This is a structural block diagram of a semantic vector map robust construction system 200 provided in an embodiment of the present application.

[0066] like Figure 2 As shown, the semantic vector map robust construction system 200 mainly includes: Image acquisition module 201, used to acquire multi-exposure original images; A preprocessing module 202 is used to preprocess the multi-exposure original image to obtain a high dynamic range image; A feature extraction module 203 is used to extract and enhance BEV features of the high dynamic range image to obtain multi-scale BEV features and enhanced BEV features; A mask processing module 204 is configured to process the enhanced BEV features to obtain a hybrid query, and process the hybrid query and the multi-scale BEV features through a multi-layer mask-activated instance decoder to obtain an embedded query and a coordinate query, wherein the coordinate query includes lane point coordinates; The map generation module 205 is used to process the enhanced BEV features through the mask feature construction unit to obtain binary mask features, determine local block features based on coordinate query, and adjust the lane point coordinates based on the local block features, embedding query, and multi-layer perceptron to obtain a semantic vectorized map.

[0067] As an optional implementation manner of this embodiment, the preprocessing module 202 is specifically used to preprocess the multi-exposure original image to obtain a high dynamic range image, including: normalizing and gamma correcting the multi-exposure original image to obtain a low dynamic range image; stitching each low dynamic range image with the corresponding multi-exposure original image in the channel dimension using a channel stitching technology to obtain a first stitched image; aligning the first stitched image through an optical flow alignment unit to obtain an aligned image; stitching the aligned images in the channel dimension to obtain a second stitched image; passing the second stitched image through multiple high-frequency enhancement units in series to obtain initial high-frequency features of different frequencies output by the multiple high-frequency enhancement units; fusing the multiple initial high-frequency features by element-by-element addition to obtain a fused high-frequency feature; extracting shallow features of a reference frame through a convolution block, where the reference frame is the multi-exposure original image with the lowest exposure; stitching the fused high-frequency feature with the shallow feature in the channel dimension, and further fusing them through 3×3 convolution to obtain a first fused feature; using 1×1 convolution to map the number of channels of the first fused feature to the target image dimension to obtain a high dynamic range image.

[0068] As an optional implementation of this embodiment, the high-frequency enhancement unit includes an average pooling layer and a highest pooling layer, and the preprocessing module 202 is specifically used to pass the second stitched image through multiple high-frequency enhancement units in series to obtain initial high-frequency features of different frequencies output by multiple high-frequency enhancement units, including: if the high-frequency enhancement unit currently being processed is the first high-frequency enhancement unit, then down-sampling the second stitched image through the average pooling layer to obtain a first low-frequency feature; if the high-frequency enhancement unit currently being processed is not the first high-frequency enhancement unit, then down-sampling the initial high-frequency feature obtained by the previous high-frequency enhancement unit through the average pooling layer to obtain a first low-frequency feature; processing the first low-frequency feature through a preset step to obtain a second fused feature; down-sampling the second fused feature through the highest pooling layer to obtain a second low-frequency feature; processing the second low-frequency feature through a preset step to obtain a first fused feature. The low-frequency features are processed to obtain a third fused feature; the third fused feature is processed through multiple serial convolution enhancement blocks to obtain an initial high-frequency feature; the first low-frequency feature is processed through preset steps to obtain a second fused feature, including: upsampling the first low-frequency feature through bilinear interpolation to obtain the original feature; subtracting the first low-frequency feature from the original feature to obtain a first high-frequency feature; globally enhancing the first high-frequency feature through a self-attention mechanism to obtain an enhanced high-frequency feature; processing the enhanced high-frequency feature through a multi-branch block to obtain a first processed data; processing the first low-frequency feature through three multi-branch blocks to obtain a second processed data; splicing the first processed data and the upsampled second processed data in the channel dimension to obtain a spliced feature; fusing the spliced feature through channel attention and convolution to obtain a second fused feature.

[0069] As an optional implementation of this embodiment, the feature extraction module 203 is specifically used to extract and enhance BEV features of high dynamic range images to obtain multi-scale BEV features and enhanced BEV features, including: extracting 2D features of high dynamic range images through a shared CNN backbone; using deformable attention to extract 2D features into BEV representation to obtain BEV features; processing BEV features through a 3-layer enhanced multi-level neck network to obtain multi-scale BEV features; upsampling the multi-scale BEV features through bilinear interpolation, aligning the multi-scale BEV features, and the aligned multi-scale BEV features have the same resolution as the BEV features; aggregating the aligned multi-scale BEV features through a series of 3×3 convolutional layers to obtain enhanced BEV features.

[0070] As an optional implementation of this embodiment, the mask processing module 204 is specifically used to process the enhanced BEV features to obtain a hybrid query, and process the hybrid query and the multi-scale BEV features through an instance decoder activated by multiple layers of masks to obtain an embedded query and a coordinate query, where the coordinate query includes lane point coordinates, including: performing basic convolution on the enhanced BEV features and applying a sigmoid function to obtain a set of instance segmentation mask maps composed of instance masks; generating a lane query based on the instance segmentation mask map and the enhanced BEV features; generating a point query based on a set of predefined learnable weights; fusing the point query with the lane query through a multi-layer perceptron to obtain a hybrid query; constructing an L-layer instance decoder through a multi-scale deformable DETR structure; and determining the embedded query and coordinate query through the instance decoder and the hybrid query.

[0071] As an optional implementation manner of this embodiment, the map generation module 205 is specifically used to process the enhanced BEV features through the mask feature construction unit to obtain binary mask features, determine local block features based on coordinate queries, and adjust the lane point coordinates based on the local block features, embedding queries and multi-layer perceptrons to obtain a semantic vectorized map, including: performing basic convolution on the enhanced BEV features and applying a sigmoid function to obtain a binary mask; constructing binary mask features based on the binary mask; determining the lane point coordinates based on the coordinate query; determining a square target box with a preset area centered on the lane point coordinates; determining local block features through the square target box, a specific area alignment method and a preset function; refreshing the feature query based on the embedding query, local block features and multi-sensor head attention; and regressing the lane point coordinates and classification scores from the feature query through the multi-layer perceptron branch to obtain a semantic vectorized map.

[0072] As an optional implementation of this embodiment, the map generation module 205 is specifically used to construct a binary mask feature based on the binary mask, including: encrypting the dimension of the binary mask through a 1×1 convolution kernel; cascading the encrypted binary mask, enhanced BEV features and a 2-channel normalized position grid, and performing a convolution operation on the cascaded result to obtain a binary mask feature.

[0073] In one example, the module in any of the above systems can be one or more integrated circuits configured to implement the above methods, such as: one or more application specific integrated circuits (ASICs), or, one or more digital signal processors (DSPs), or, one or more field programmable gate arrays (FPGAs), or a combination of at least two of these integrated circuit forms.

[0074] For another example, when the modules in the system can be implemented in the form of a processing element scheduler, the processing element can be a general-purpose processor, such as a central processing unit (CPU) or other processor capable of calling programs. For another example, these modules can be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0075] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0076] Figure 3 This is a structural block diagram of an electronic device 300 provided in an embodiment of the present application.

[0077] like Figure 3 As shown, the electronic device 300 includes a processor 301 and a memory 302 , and may further include an information input / information output (I / O) interface 303 , one or more communication components 304 , and a communication bus 305 .

[0078] The processor 301 is used to control the overall operation of the electronic device 300 to complete all or part of the steps of the robust semantic vector map construction method described above. The memory 302 is used to store various types of data to support the operation of the electronic device 300. Such data may include, for example, instructions for any application or method operating on the electronic device 300, as well as application-related data. The memory 302 may be implemented by any type of volatile or non-volatile storage device, or a combination thereof, such as one or more of static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0079] The I / O interface 303 provides an interface between the processor 301 and other interface modules, which may be a keyboard, a mouse, buttons, etc. These buttons may be virtual buttons or physical buttons. The communication component 304 is used for wired or wireless communication between the electronic device 300 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G or 4G, or a combination of one or more thereof, therefore, the corresponding communication component 304 may include: Wi-Fi components, Bluetooth components, NFC components.

[0080] The electronic device 300 can be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to execute the robust semantic vector map construction method given in the above embodiment.

[0081] Communication bus 305 may include a path for transmitting information between the aforementioned components. Communication bus 305 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, for example. Communication bus 305 may be divided into an address bus, a data bus, a control bus, and the like.

[0082] The electronic device 300 may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., as well as fixed terminals such as digital TVs, desktop computers, etc., and may also be servers, etc.

[0083] The present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned method for robustly constructing a semantic vector map are implemented.

[0084] The computer-readable storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc., which can store program codes.

[0085] The present application also provides a computer program product, which stores a computer program or instructions. When the computer program or instructions are executed by a processor, the steps of the above-mentioned method for robustly constructing a semantic vector map are implemented.

[0086] The computer program product may be written in any combination of one or more programming languages to implement the program code for performing the operations of the embodiments of the present application, including object-oriented programming languages such as Java, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0087] The terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed or inherent to such process, method, article, or apparatus.

[0088] The above description is merely a preferred embodiment of the present application and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of application involved in this application is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the aforementioned application concept. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions applied for in this application.

Claims

1. A robust construction method for semantic vector maps, characterized in that: include: Get multi-exposure original images; Preprocessing the multi-exposure original image to obtain a high dynamic range image; Extracting and enhancing BEV features from the high dynamic range image to obtain multi-scale BEV features and enhanced BEV features; Processing the enhanced BEV features to obtain a hybrid query, processing the hybrid query and the multi-scale BEV features through a multi-layer mask-activated instance decoder to obtain an embedded query and a coordinate query, wherein the coordinate query includes lane point coordinates; The enhanced BEV features are processed by a mask feature construction unit to obtain binary mask features, local block features are determined based on the coordinate query, and the lane point coordinates are adjusted based on the local block features, the embedding query, and a multi-layer perceptron to obtain a semantic vectorized map.

2. The method according to claim 1, characterized in that The preprocessing of the multi-exposure original image to obtain a high dynamic range image includes: performing normalization and gamma correction processing on the multi-exposure original image to obtain a low dynamic range image; Stitching each of the low dynamic range images with the corresponding multi-exposure original image in a channel dimension using a channel stitching technology to obtain a first stitched image; Aligning the first stitched images using an optical flow alignment unit to obtain an aligned image; Stitching the aligned images in a channel dimension to obtain a second stitched image; Passing the second stitched image through a plurality of high-frequency enhancement units connected in series to obtain initial high-frequency features of different frequencies output by the plurality of high-frequency enhancement units; Fusing the multiple initial high-frequency features by element-by-element addition to obtain a fused high-frequency feature; extracting shallow features of a reference frame through a convolution block, wherein the reference frame is the multi-exposure original image with the lowest exposure; The fused high-frequency features and the shallow features are concatenated in the channel dimension, and further fused through 3×3 convolution to obtain a first fused feature; A 1×1 convolution is used to map the number of channels of the first fusion feature to the target image dimension to obtain the high dynamic range image.

3. The method according to claim 2, characterized in that The high-frequency enhancement unit includes an average pooling layer and a highest pooling layer. The second spliced image is passed through multiple high-frequency enhancement units connected in series to obtain initial high-frequency features of different frequencies output by the multiple high-frequency enhancement units, including: If the high-frequency enhancement unit currently being processed is the first high-frequency enhancement unit, downsampling the second stitched image through the average pooling layer to obtain a first low-frequency feature; if the high-frequency enhancement unit currently being processed is not the first high-frequency enhancement unit, downsampling the initial high-frequency feature obtained by the previous high-frequency enhancement unit through the average pooling layer to obtain the first low-frequency feature; Processing the first low-frequency feature through a preset step to obtain a second fusion feature; Downsampling the second fused feature through the highest pooling layer to obtain a second low-frequency feature; Processing the second low-frequency feature through the preset steps to obtain a third fusion feature; Processing the third fused feature through a plurality of serially connected convolution enhancement blocks to obtain the initial high-frequency feature; The step of processing the first low-frequency feature through a preset step to obtain a second fusion feature includes: Upsampling the first low-frequency feature by bilinear interpolation to obtain the original feature; Subtracting the first low-frequency feature from the original feature to obtain a first high-frequency feature; Globally enhancing the first high-frequency feature through a self-attention mechanism to obtain an enhanced high-frequency feature; Processing the enhanced high-frequency features through a multi-branch block to obtain first processed data; Processing the first low-frequency feature through three multi-branch blocks to obtain second processed data; Splicing the first processed data and the second processed data obtained by upsampling in a channel dimension to obtain a splicing feature; The spliced features are fused through channel attention and convolution to obtain the second fused features.

4. The method according to claim 1, wherein The extracting and enhancing BEV features of the high dynamic range image to obtain multi-scale BEV features and enhanced BEV features includes: Extracting 2D features of the high dynamic range image through a shared CNN backbone; Extracting the 2D features into BEV representation using deformable attention to obtain BEV features; The BEV features are processed by a 3-layer enhanced multi-level neck network to obtain multi-scale BEV features; Upsampling the multi-scale BEV features by a bilinear interpolation method, and aligning the multi-scale BEV features, so that the aligned multi-scale BEV features have the same resolution as the BEV features; The enhanced BEV features are obtained by aggregating the aligned multi-scale BEV features through the series-connected 3×3 convolutional layers.

5. The method according to claim 1, characterized in that The enhanced BEV features are processed to obtain a hybrid query, and the hybrid query and the multi-scale BEV features are processed by a multi-layer mask-activated instance decoder to obtain an embedded query and a coordinate query, wherein the coordinate query includes lane point coordinates, including: Performing basic convolution on the enhanced BEV features and applying a sigmoid function to obtain a set of instance segmentation mask maps consisting of instance masks; generating a lane query based on the instance segmentation mask and the enhanced BEV features; Generate point queries based on a set of predefined learnable weights; fusing the point query and the lane query through a multi-layer perceptron to obtain a hybrid query; Construct an L-layer instance decoder through a multi-scale deformable DETR structure; An embedding query and a coordinate query are determined by the instance decoder and the hybrid query.

6. The method according to claim 1, characterized in that The enhanced BEV feature is processed by a mask feature construction unit to obtain a binary mask feature, a local block feature is determined based on the coordinate query, and the lane point coordinates are adjusted based on the local block feature, the embedding query, and a multi-layer perceptron to obtain a semantic vectorized map, including: Performing basic convolution on the enhanced BEV feature and applying a sigmoid function to obtain a binary mask; Constructing a binary mask feature based on the binary mask; determining the lane point coordinates based on the coordinate query; Determine a square target frame with a preset area centered on the lane point coordinates; Determine local block features by using the square target frame, the specific area alignment method, and the preset function; Refreshing feature query based on the embedding query, the local block feature, and multi-sensor head attention; Lane point coordinates and classification scores are regressed from the feature query through a multi-layer perceptron branch to obtain a semantic vectorized map.

7. The method according to claim 6, characterized in that The constructing a binary mask feature based on the binary mask includes: Encrypting the dimension of the binary mask by a 1×1 convolution kernel; The encrypted binary mask, the enhanced BEV feature and a 2-channel normalized position grid are cascaded, and a convolution operation is performed on the cascade result to obtain the binary mask feature.

8. An electronic device, characterized in that: comprising a processor coupled to a memory; The processor is configured to execute the computer program stored in the memory, so that the electronic device performs the method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that The method comprises a computer program or an instruction, which, when executed on a computer, causes the computer to execute the method according to any one of claims 1 to 7.

10. A computer program product, characterized in that The method comprises a computer program or instructions, which implements the method according to any one of claims 1 to 7 when executed by a processor.

Images