Vector map construction method and system based on view interactive aggregation

Abstract

The invention relates to a vector map construction method and system based on view interactive aggregation, and the method comprises the steps: obtaining a visible light image, respectively extracting the features of the visible light image in a pixel-level coordinate system and a camera coordinate system, and obtaining a fusion feature through an asynchronous mutual learning fusion module; obtaining depth instance features and geometric position features through a key point pre-selection module by the fusion features, and taking the depth instance features and the geometric position features as initial reference points; taking fusion features and key point query and instance query obtained based on a key point pre-selection module as input of a target detection algorithm, taking an obtained initial reference point as additional prompt features of the target detection algorithm, and obtaining predicted vector map element tags and positions of the vector map element tags; the predicted vector map element tags and the positions where the vector map element tags are located form a single-frame vector map, rich feature information in a historical frame and sparse features in a current frame are aggregated to enhance the sparse features of the current frame, and the vector map is further obtained through a detection head.

Description

Technical Field

[0001] The present invention relates to the technical field of map construction, and in particular to a vector map construction method and system based on view interactive aggregation. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] To achieve autonomous driving of the vehicle, map data needs to be updated in real time, that is, maps are built through algorithms. Many methods of building maps rely too much on depth information. When the depth estimation is inaccurate, the quality of map generation is greatly reduced. At the same time, because vision-based methods are sensitive to extreme conditions such as insufficient lighting and object occlusion, and to environmental factors such as rain, fog, and low light, the information in map construction is sparse, resulting in incomplete and inaccurate construction. Therefore, the existing technology begins to focus on the use of BEV (Bird Eye's View) technology in map construction methods, and then builds more accurate vectorized maps to help with the positioning and planning of autonomous vehicles.

[0004] Existing BEV map generation is mainly divided into two modes: map rasterization generation and map vectorization generation. The map rasterization method uses key feature points extracted from sensor data to build maps. However, since it is difficult to distinguish different map elements and cannot provide rich structural information, it requires a lot of post-processing work before it can be used for downstream tasks.

[0005] To solve the above problems, map vectorization methods have become the mainstream research direction for real-time construction of high-precision maps. Compared with map rasterization methods, map generation in a vectorized manner has significant advantages in updating and maintaining map information elements. It can not only restore map information during driving, but also ensure the integrity and real-time nature of the information. However, there are two significant problems with existing methods:

[0006] First, there is an incomplete problem in considering local information and global information. The traditional feature fusion method will lead to feature redundancy, especially when building vector maps. This fusion strategy may affect the distinguishability of map instances.

[0007] Second, in the process of vector map construction, there is a problem of sparse visual information features. The existing instance feature extraction methods have not been able to fully capture all relevant visual information, resulting in the problem of missing map details caused by information sparsity. Summary of the invention

[0008] In order to solve the technical problems existing in the above-mentioned background technology, the present invention provides a vector map construction method and system based on view interaction aggregation, which can correct the geometric distortion of long-distance lane lines encountered during the driving of autonomous driving vehicles, and optimize the feature sparseness problem caused by occlusion in single-frame scenes, thereby generating a more complete BEV vectorized map and improving the accuracy of positioning and decision-making in the autonomous driving system.

[0009] In order to achieve the above object, the present invention adopts the following technical solution:

[0010] A first aspect of the present invention provides a vector map construction method based on view interactive aggregation, comprising the following steps:

[0011] Obtain a visible light image, and extract features of the visible light image in a pixel-level coordinate system and a camera coordinate system respectively;

[0012] The features in the pixel-level coordinate system and the features in the camera coordinate system are fused through an asynchronous mutual learning fusion module to obtain fused features;

[0013] The fused features are passed through the key point pre-selection module to obtain deep instance features and geometric position features, which are used as initial reference points. The fused features, key point queries and instance queries obtained based on the key point pre-selection module are used as inputs to the target detection algorithm. The obtained initial reference points are used as additional prompt features of the target detection algorithm to obtain the predicted vector map element labels and their locations.

[0014] The predicted vector map element labels and their locations form a single-frame vector map. The rich feature information in the historical frames is aggregated with the sparse features in the current frame to enhance the sparse features of the current frame, and the vector map is further obtained through the detection head.

[0015] Furthermore, the asynchronous mutual learning fusion module combines the features F extracted in the pixel-level coordinate system pixel The decoded perspective view feature F in the camera coordinate system p Mutual learning is carried out. Specifically, in the first N rounds of training, the features extracted in the pixel-level coordinate system are selected as the teacher of the preliminary map features, and the spatial loss is calculated in the camera coordinate system for supervision; when the number of training times reaches the set boundary value, the learned preliminary map features are used as the teacher of the features extracted in the pixel-level coordinate system, and the foreground and background of the feature map are predicted to achieve mutual learning between the two and obtain fused features.

[0016] Furthermore, the asynchronous mutual learning fusion module is shown in the following formula:

[0017]

[0018] Among them, LRDM represents the asynchronous mutual learning model, in LRDM(x1,x2), x1 represents the guided feature, x2 represents the feature to be learned, epoch is the number of training times, and F pixel is the feature extracted in the pixel-level coordinate system, F p It is the decoded perspective view feature in the camera coordinate system.

[0019] Furthermore, the key point pre-selection module selects the initial vector points from the sparse instance map, obtains a point query by mixing the semantic features and position features of each vector point, and then performs multi-layer optimization in the decoder.

[0020] Furthermore, the key point pre-selection module uses the BEV feature Generate coarse instance features about vector points Obtaining instance maps through the segmentation head Where N represents the number of categories of pre-selected instances.

[0021] Further, from the instance mapping The key vector points are selected and the importance of the key points is learned. The key points with high weights are used as the initial vector points, and their geometric positions are output as the pre-selected vector points Pres xy ∈N×P×2, N represents the number of instance categories, and P represents the number of points in that category.

[0022] Furthermore, by preselecting the depth instance feature Depth of the vector point instfea Back Projection Rev proj The initial position feature map is then transferred to the original position feature map to obtain the corresponding depth instance feature, and the standardized pixel coordinates of the vector points in the instance map are encoded to obtain the geometric position feature.

[0023] Furthermore, the obtained deep instance feature Depth instfea and geometric position feature Pos instfea , which is the feature of the final pre-selected key point, and is used as the initial reference point P key .

[0024] Furthermore, the result of decoding each map instance is used as a point query Will Initialize a random variable as the instance object Query, and query Q with the selected key points key Query by Example I Perform the addition operation as the Query input Q of the target detection algorithm tot =Q key +Q I , embed BEV into Fins As the Key / Value input of the target detection algorithm. The initial reference point P key It is input into the target detection algorithm as an additional hint feature, and a linear layer combined with a normalization layer and a RELU activation function is used as the task head to predict the label and location of the instance, that is, to obtain the predicted vector map element label and location.

[0025] A second aspect of the present invention provides a system for implementing the above method, comprising:

[0026] The data acquisition and preprocessing module is configured as follows:

[0027] The feature extraction module is configured to: acquire a visible light image, and extract features of the visible light image in a pixel-level coordinate system and a camera coordinate system respectively;

[0028] The view fusion module is configured as follows: the features in the pixel-level coordinate system and the features in the camera coordinate system are fused by an asynchronous mutual learning fusion module;

[0029] The feature decoding module is configured to: fuse the features through the key point pre-selection module to obtain the deep instance features and geometric position features, and use them as the initial reference points; use the fused features, the key point query and the instance query obtained based on the key point pre-selection module as the input of the target detection algorithm, and use the obtained initial reference points as additional prompt features of the target detection algorithm to obtain the predicted vector map element labels and their locations;

[0030] The feature enhancement module is configured to: form a single-frame vector map with the predicted vector map element labels and their locations, aggregate the rich feature information in the historical frames with the sparse features in the current frame to enhance the sparse features of the current frame, and further obtain the vector map through the detection head.

[0031] A third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in constructing a vector map based on view interaction aggregation as described above.

[0032] A fourth aspect of the present invention provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in constructing a vector map based on view interaction aggregation as described above are implemented.

[0033] A fifth aspect of the present invention provides a computer program product, including a computer program, which, when executed by a processor, implements the steps in constructing the above-mentioned vector map based on view interaction aggregation.

[0034] Compared with the prior art, one or more of the above technical solutions have the following beneficial effects:

[0035] 1. One of the main methods to achieve BEV semantic mapping is to design a Transformer for deep learning of images. This Transformer can display BEV mapping with depth estimation. When designing a view Transformer for deep learning in a pixel coordinate system, the early learning speed is slow, but once the training is completed, the effect is significant. In addition, another mainstream method to achieve BEV semantic mapping is to display depth estimation of the BEV mapping with depth estimation through a perspective view in a camera coordinate system, and assign an assumed depth to each pixel. However, this method distorts the feature extraction of the same object in the image due to the perspective effect of near large and far small. Therefore, through the asynchronous mutual learning fusion module, combined with the advantages of the pixel coordinate system and the camera coordinate system, the learning process of the BEV semantic mapping is optimized to obtain a complete lane line representation from a bird's-eye view.

[0036] 2. A key point pre-selection module is designed for the area of ​​concern of feature decoding to pay more attention to the vectorized features of the target area during decoding, and the accuracy of instance prediction is improved by pre-selecting unclassified features.

[0037] 3. Through instance feature aggregation enhancement, we can address the challenge of sparse target lane line features caused by occlusion problems in single-frame scenarios. We can aggregate the correlation of sparse features in adjacent frames of the time series, and enhance the expression of sparse features in the current frame by utilizing the richer features extracted from the previous frame, making the sparse instance features of the current frame richer. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0039] Figure 1 It is an overall flow chart of a BEV vectorized map construction process based on view interaction aggregation in an autonomous driving scenario provided by one or more embodiments of the present invention;

[0040] Figure 2 is a schematic diagram of the structure of an asynchronous mutual learning fusion module provided by one or more embodiments of the present invention;

[0041] Figure 3 is a schematic diagram of the structure of a key point pre-selection module provided by one or more embodiments of the present invention;

[0042] Figure 4is a schematic diagram of the structure of an example feature aggregation enhancement module provided by one or more embodiments of the present invention;

[0043] Figure 5 A BEV vectorized map construction system based on view interaction aggregation in an autonomous driving scenario provided by one or more embodiments of the present invention;

[0044] Figure 6 It is a schematic diagram of the structure of an electronic device provided by one or more embodiments of the present invention. DETAILED DESCRIPTION

[0045] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0046] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0047] Embodiment 1:

[0048] The vector map construction method based on view interaction aggregation includes the following steps:

[0049] Acquire visible light images of the area to be mapped in real time;

[0050] Based on the obtained visible light image, the feature extraction module is used to extract features of the image in the pixel-level coordinate system and the camera coordinate system, and then the data in the camera coordinate system is preliminarily decoded to obtain preliminary map features.

[0051] The asynchronous mutual learning fusion module (Long-range distortion rectification Module, LDRM) is used to learn the features extracted in the pixel-level coordinate system and the decoded perspective view features in the camera coordinate system. Specifically:

[0052] In the first N rounds of training, since the features in the pixel-level coordinate system have certain prior knowledge and stronger learning ability, they can transfer the learned knowledge to the features with relatively weaker ability. Therefore, the features extracted in the pixel-level coordinate system are selected as the teachers of the preliminary map features, and the spatial loss is calculated in the camera coordinate system for supervision. When the epoch reaches the set boundary value, the learned preliminary map features are used as the teachers of the features extracted in the pixel-level coordinate system. By predicting the foreground and background of the feature map, the mutual learning between the two is realized, and finally the effect of view fusion is achieved.

[0053] In the feature decoding module, the area that Deformable DETR focuses on in the traditional way is greatly restricted by the input features. The Key Point Preselection Module (KPPM) is used to select the initial vector points from the sparse instance map. By mixing the semantic features and position features of each vector point, a point query is obtained, which is then optimized through multiple layers in the decoder.

[0054] Using the Instance Feature Aggregation Enhancement Module (IFAEM), based on the instance features in the historical frames and the current frame, with the help of instance boxes, labels and supervised instance scores, a consistent learning strategy is formulated to aggregate the rich feature information in the historical frames with the sparse features in the current frame to enhance the sparse features of the current frame. Considering that different labels have unique clustering spaces, positive and negative samples are selected independently under different labels to ensure the extensiveness of contrastive learning.

[0055] like Figure 1 As shown, this embodiment takes the nuScenes dataset and the Argoverse2 dataset as examples to provide a BEV vectorized map construction method based on view interaction aggregation in an autonomous driving scenario, which specifically includes the following processing procedures:

[0056] Step 1: Data loading: Obtain the RGB image data of the map area to be generated in real time as model input.

[0057] In a specific implementation, visible light images of the area to be mapped are collected in real time based on vehicle-mounted sensors, wherein the vehicle-mounted sensors are integrated with camera devices and lidar devices.

[0058] Step 2: Feature extraction: Extract features of the input camera data in the pixel coordinate system and the camera coordinate system respectively. The features extracted in the camera coordinate system are initially encoded and decoded, and the results obtained are used as the input for subsequent feature fusion.

[0059] In the specific implementation, the RGB image collected by the camera is used as input. Represents RGB image input, Represents a three-dimensional matrix; W is the width; H is the width; C represents the number of channels. Feature extraction F is performed in the pixel-level coordinate system respectively pixel And the feature extraction F in the camera coordinate system camera , the features extracted in the camera coordinate system are initially decoded to obtain the global features in the perspective view:

[0060]

[0061] in θ i (·) represents the fully connected layer at the i-th layer.

[0062] Step 3: View fusion: The extracted pixel features and the preliminary results in the camera coordinate system are asynchronously fused through the asynchronous mutual learning module (LDRM) to obtain the feature fusion results of the depth information and the preliminary decoding under the perspective view.

[0063] Specifically, Figure 2 As shown in the figure, the asynchronous mutual learning module (LRDM) interactively learns the two input features to fuse the features extracted in different coordinate systems.

[0064] Asynchronous mutual learning module LRDM: The feature F extracted in the pixel-level coordinate system pixel The decoded perspective view feature F in the camera coordinate system p First, in the first N epochs of training, since the features in the pixel-level coordinate system have certain prior knowledge and stronger learning ability, they can transfer the learned knowledge to the features with relatively weaker ability, so F is selected. pixel As F p The teacher is used to calculate the spatial loss in the camera coordinate system for supervision. When the epoch reaches the set boundary value, F p As F pixel The teacher predicts the foreground and background of the feature map to achieve mutual learning between the two and finally achieve the effect of view fusion. LRDM represents the asynchronous mutual learning model, LRDM (x1, x2) where x1 represents the guiding feature and x2 represents the feature to be learned.

[0065]

[0066] Step 4: Feature decoding: Decode the BEV features using the DETR framework to obtain map elements of the vectorized map. The key point Query selection uses the key point pre-selection module (KPPM) of this embodiment.

[0067] Specifically, Figure 3 As shown, the key point pre-selection module pre-selects unclassified features to improve the accuracy of instance prediction.

[0068] Key Point Pre-selection Module KPPM: Deformable DETR is used for decoding, where the decoder input contains a fusion feature obtained from the asynchronous mutual learning module as the BEV embedding F ins , an example query (where N is the number of instance categories and C is the number of feature channels), the initial reference point (N p Represents a fixed number of vector points in each instance) and key point query Query

[0069] In this embodiment, BEV embedding is the BEV feature extracted by the encoder, that is, the fused feature obtained in step three. Since this feature needs to be used as an embedding to assist the subsequent Transformer in understanding Queri es, it is named BEV embedding in this embodiment.

[0070] For the selection of key point queries, we first use the BEV feature Generate coarse instance features about vector points Use a simple segmentation head to get instance maps Where N represents the number of categories of pre-selected instances. Then a form of network mapping can be constructed to represent the instance mapping:

[0071]

[0072]

[0073] in It is a complete three-layer ResNet structure. Is a split header structure.

[0074] For the segmented instance map, the key vector points are selected by key point selection, and a weight module is proposed to learn the importance of key points. The key points with high weights are used as the initial vector points, and their geometric positions are output as pre-selected vector points Pres xy ∈N×P×2 (N represents the number of instance categories, and P represents the number of points in that category):

[0075] Weight map =W(Map ins );

[0076] Pres xy =Norm(TopK(Flatten(Weight map ));

[0077] Where W(·) represents the weight module to be learned, TopK is used to select the position with the highest score and output the position coordinates. Norm is used here to normalize the coordinates to avoid inconsistency in the coordinate system. instfea Back Projection Rev proj To the initial position feature map, we get the deep instance features of these points:

[0078] Pres′ xy =Pres xy [h, w] T ;

[0079] Pres″ xy =Pres′ xy [0]·Pres′ xy [1]*w;

[0080] Depth instfea =Rev proj (Flatten(F ins ), Pres″ xy );

[0081] Due to Depth instfea Due to the lack of relative geometric features, two MLP layers M2 are used to encode the normalized pixel coordinates of vector points in the instance map. The geometric position features are represented as follows:

[0082] Pos instfea =M2(Pres xy )

[0083] The final pre-selected key point features are composed of the deep instance features Depth instfea and geometric position feature Pos instfea constituted, which is used as the initial reference point P key :

[0084] P key =Depth instfea +Pos instfea

[0085] At the same time, the result of decoding each map instance is used as a point query Will Initialize a random variable as the instance object's Query, and finally query Q with the selected key points key Query by Example I Perform the addition operation and use it as the query input Q of Deformable DETR tot =Q key +Q I , embed BEV into F ins As the Key / Value input of Deformable DETR. Initial reference point P key As an additional hint feature, it is input into Deformable DET R to help instance recognition and prediction. Finally, a linear layer combined with a normalization layer and a RELU activation function is used as the task head to predict the instance label and location.

[0086] Instances, or vector map elements, refer to specific, independent map elements or objects. These instances have clear definitions and attributes in vector maps and are used to describe key information such as roads, lanes, traffic signs, obstacles, etc.

[0087] The Deformable DETR in this embodiment is a target detection algorithm based on Transformer and deformable convolution.

[0088] Step 5: Feature enhancement: Due to the sparse features in single-frame occlusion scenes, weak features exist after decoding. The weak features are sent to the designed feature aggregation module (IFAEM) for correlation aggregation to generate enhanced instance features to improve the quality of the vectorized map.

[0089] Specifically, Figure 4 As shown, the instance feature aggregation module aggregates and enhances the sparse features in the current frame and the features in the historical frames to generate enhanced instance features to improve the quality of the vectorized map.

[0090] Instance Feature Aggregation Enhancement Module IFAEM: Since the instance features aggregated from KPPM are the output of single-frame perception information, the instance features extracted from a single-frame occluded scene are very sparse, so the single-frame instance feature results processed by KPPM are used as the input of IFAEM. Based on the instance features in the historical frames and the current frame, the IFAEM module formulates a consistency learning strategy with the help of instance boxes, labels, and supervised instance scores, and aggregates the rich feature information in the historical frames with the sparse features in the current frame to enhance the sparse features of the current frame. Considering that there are unique clustering spaces between different labels, positive and negative samples are selected independently under different labels to ensure the extensiveness of contrastive learning. The aggregated features with higher instance scores in each semantic label are sent to the contrast head, which is generated by the comprehensive evaluation of instance and location accuracy under the supervision of the ground truth (GT), so that the "v" embedding can be obtained as contrastive learning. Since the number of queries is greater than the number of instances in each frame, the number of "v" is designed as a dynamic norm. For each label, a fixed maximum value "v" is set.

[0091] The selection mechanism for positive and negative samples is formulated as follows: In adjacent frames of a time series, changes in the dynamic environment may lead to feature differences between instances with similar positions. In particular, due to occlusion, some instance features are sparse. However, through the learning process of positive samples, the sparse feature expression can be enhanced from relatively rich features. Therefore, the instance features History in the history frames within a certain range around the "v" instance are selected. inst As a positive sample T Negative sample F is selected from negative labels, allowing features of instances with the same label to be pulled closer together. Negative labels refer to categories other than "v" itself. The number of negative samples for each negative label is fixed. The instance sample with the highest score in the current frame is selected as the negative sample to improve learning efficiency.

[0092]

[0093]

[0094]

[0095] Label T and Label F is the selected label and k is a fixed value.

[0096] Step 6: Send the features output by the enhancement module to the detection head to obtain the final detection results.

[0097] In the current scheme, considering the negative impact of image geometric distortion and the positive and negative sample errors in instance consistency learning, the application of the present invention may cause prediction errors in actual map construction tasks. The present invention designs a global loss function to simultaneously consider the negative impact of both through the back propagation of the model, as follows:

[0098] Considering that geometric distortion will be brought into the fusion process of the features in the pixel coordinate system and the decoded results in the camera coordinate system if it is not checked, the difference between the decoded perspective view features in the camera coordinate system and the ground truth is calculated using the cross entropy loss method to ensure the accuracy of the generated BEV roadmap and reduce the impact of any geometric distortion that may occur:

[0099]

[0100] Where y represents the number of cameras, L xy is the cross entropy loss.

[0101] Considering the errors between positive and negative samples in instance consistency learning, since the selection of positive and negative samples can be regarded as training a classification model, the performance of the model in correctly selecting positive and negative samples is improved by minimizing the difference between the model prediction probability distribution and the true label probability distribution:

[0102] Loss IFAEM =log[1+∑ cls ∑ Sam +∑ Sam -exp(v cls *Sam - -v cls *Sam + )];

[0103] Among them, v cls Represents the dynamic norm of each category, Sam + and Sam - Represents positive samples and negative samples.

[0104] Therefore, a global loss function Loss is proposed for this embodiment to solve the positive and negative sample errors of geometric distortion and instance consistency learning, so as to ensure that the accuracy of map prediction generation can be improved. The specific expression is as follows:

[0105] Loss=Loss LRDM +Loss IFAEM .

[0106] One of the main methods to achieve BEV semantic mapping is to design a Transformer for deep learning of images, which can display the BEV map of depth estimation. When designing a view Transformer for deep learning in a pixel coordinate system, there is a challenge: the early learning speed is slow, but once the training is completed, the effect is significant. Another mainstream method to achieve BEV semantic mapping is to display the depth estimation of the BEV map with depth estimation through the perspective view in the camera coordinate system, and assign an assumed depth to each pixel. However, this method also brings some problems, such as the feature of near big and far small caused by the perspective effect, which distorts the feature extraction of the same object in the image.

[0107] Therefore, this embodiment provides a BEV vectorized map construction method based on view interaction aggregation in an autonomous driving scenario, and proposes an asynchronous mutual learning fusion module, which aims to optimize the learning process of BEV semantic mapping by combining the advantages of the pixel coordinate system and the camera coordinate system, and obtain a complete lane line representation from a bird's-eye view perspective.

[0108] At the same time, a key point pre-selection module is designed for the area of ​​concern for feature decoding to pay more attention to the vectorized features of the target area during decoding. Currently, in the process of making vectorized maps, the generation of map element instances depends on the DETR architecture, which can decode these instances from BEV features. These initial queries are randomly generated, and all sampling points use a unified learning dimension. After training with a large amount of data, the greatest common factor in the training set is selected as the initial value of the query. However, for those vector point instances far away from the center point, the regression learning process requires more iterations to converge. This makes the current initialization method inefficient when processing high-definition maps containing multiple shape instances. Therefore, in order to improve efficiency and make the target area more concerned, this embodiment designs a key point pre-selection module, which can perform differentiated key point selection according to different instance features, so as to more effectively aggregate instance features, and the module can be embedded in any map generation algorithm with Deformable DETR as a decoder, with good generalization.

[0109] A new component for instance feature aggregation enhancement is also designed to address the challenge of sparse target lane line features caused by occlusion problems in single-frame scenarios. It can correlate the sparse features in adjacent frames of the time series and enhance the expression of sparse features in the current frame by utilizing the richer features extracted from the previous frame, making the sparse instance features of the current frame richer.

[0110] The method in the above embodiment can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor. The software module can be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware. To avoid repetition, it will not be described in detail here.

[0111] Embodiment 2:

[0112] like Figure 5 As shown, a system for implementing the above method includes:

[0113] The feature extraction module is configured to: acquire a visible light image, and extract features of the visible light image in a pixel-level coordinate system and a camera coordinate system respectively;

[0114] The view fusion module is configured as follows: the features in the pixel-level coordinate system and the features in the camera coordinate system are fused through asynchronous mutual learning to obtain fused features;

[0115] The feature decoding module is configured to: fuse the features through the key point pre-selection module to obtain the deep instance features and geometric position features, and use them as the initial reference points; query Q with the fused features and the key point obtained based on the key point pre-selection module key Query by Example I , as the input of the target detection algorithm, the obtained initial reference point is used as an additional prompt feature of the target detection algorithm to obtain the predicted vector map element label and its location;

[0116] The feature enhancement module is configured to: form a single-frame vector map with the predicted vector map element labels and their locations, aggregate the rich feature information in the historical frames with the sparse features in the current frame to enhance the sparse features of the current frame, and further obtain the vector map through the detection head.

[0117] It can be understood that the system is consistent with the method in the above embodiment, and its specific details are described in detail in the method embodiment, so they are not repeated here.

[0118] Embodiment three:

[0119] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps in the vector map construction based on view interaction aggregation in the above-mentioned embodiment 1 are implemented.

[0120] In this embodiment, if Figure 6 As shown, the design of the processor should be considered flexible and diverse to adapt to different application requirements. Specifically, the processor can be configured to include one or more cores, such as a single-core processor, a quad-core processor, an octa-core processor, etc., to provide different degrees of computing power. In terms of hardware implementation, the processor can be based on a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, as well as discrete gate or transistor logic devices and discrete hardware components. The choice of these hardware forms depends on specific performance requirements and cost-effectiveness considerations.

[0121] Furthermore, the processor architecture may include a combination of a main processor and a coprocessor. The main processor, commonly referred to as a central processing unit (CPU), is responsible for handling computing tasks when the device is awake. A coprocessor is a low-power processor specifically designed to process data in standby mode to optimize energy consumption and extend the life of the device. In some embodiments, the processor may be integrated with a graphics processing unit (GPU), which is responsible for rendering and drawing content required for the display screen. This integrated design helps improve the efficiency of graphics processing and may reduce the overall power consumption of the system.

[0122] Embodiment 4:

[0123] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of constructing a vector map based on view interaction aggregation in the first embodiment described above are implemented.

[0124] In this embodiment, the memory is designed to provide an efficient and reliable data storage solution. The memory system consists of one or more computer-readable storage media, which should have non-transient characteristics to ensure the persistence of data. Specifically, the memory configuration includes high-speed random access memory (RAM) and non-volatile memory (NVM), the latter of which can be further subdivided into disk storage devices and flash storage devices, such as solid-state drives (SSDs) and USB flash drives. The main function of the memory is to store computer program codes, which, after being loaded and executed by the processor, can implement the relevant steps of the method performed by the electronic device side disclosed in any of the aforementioned embodiments. In addition, the memory is also responsible for storing resources such as operating systems, applications, and data, which can be short-term storage or permanent storage. The choice of storage method depends on the frequency of use and access requirements of the data. The choice of operating system should cover mainstream options, including but not limited to Windows, Linux, and Android, to meet the needs of different users and application scenarios. Such a design not only provides flexibility, but also ensures compatibility with the existing software ecosystem, thereby enhancing the applicability and user-friendliness of electronic devices. Through this comprehensive storage solution, this embodiment aims to achieve efficient management of data while maintaining the stability and response speed of the system.

[0125] Embodiment five:

[0126] This embodiment provides a computer program product, including a computer program. When the computer program is executed by a processor, the steps in the above-mentioned embodiment 1 of constructing a vector map based on view interaction aggregation are implemented.

[0127] The steps involved in the above embodiments 2 to 5 correspond to those in embodiment 1. For the specific implementation, please refer to the relevant description of embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood to include any medium that can store, encode or carry an instruction set for execution by a processor and enable the processor to execute any method in the present invention.

[0128] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A vector map construction method based on view interaction aggregation, characterized by: The following steps are involved: Obtain a visible light image, and extract features of the visible light image in a pixel-level coordinate system and a camera coordinate system respectively; The features in the pixel-level coordinate system and the features in the camera coordinate system are fused through an asynchronous mutual learning fusion module to obtain fused features; The fusion features are passed through the key point pre-selection module to obtain the deep instance features and geometric position features, which are used as the initial reference points; The fused features, key point query and instance query obtained based on the key point pre-selection module are used as the input of the target detection algorithm. The obtained initial reference point is used as an additional prompt feature of the target detection algorithm to obtain the predicted vector map element label and its location. The predicted vector map element labels and their locations form a single-frame vector map. The rich feature information in the historical frames is aggregated with the sparse features in the current frame to enhance the sparse features of the current frame, and the vector map is further obtained through the detection head.

2. The vector map construction method based on view interactive aggregation as claimed in claim 1, characterized in that: The asynchronous mutual learning fusion module combines the features F extracted in the pixel-level coordinate system pixel The decoded perspective view feature F in the camera coordinate system p Mutual learning is carried out. Specifically, in the first N rounds of training, the features extracted in the pixel-level coordinate system are selected as the teacher of the preliminary map features, and the spatial loss is calculated in the camera coordinate system for supervision; when the number of training times reaches the set boundary value, the learned preliminary map features are used as the teacher of the features extracted in the pixel-level coordinate system, and the foreground and background of the feature map are predicted to achieve mutual learning between the two and obtain fused features.

3. The vector map construction method based on view interactive aggregation as claimed in claim 1, characterized in that: The asynchronous mutual learning fusion module is shown in the following formula: Among them, LRDM represents the asynchronous mutual learning model, in LRDM(x1,x2), x1 represents the guided feature, x2 represents the feature to be learned, epoch is the number of training times, and F pixel is the feature extracted in the pixel-level coordinate system, F p It is the decoded perspective view feature in the camera coordinate system.

4. The vector map construction method based on view interactive aggregation as claimed in claim 1, characterized in that: The key point pre-selection module selects the initial vector points from the sparse instance map, obtains a point query by mixing the semantic features and position features of each vector point, and then performs multi-layer optimization in the decoder.

5. The vector map construction method based on view interactive aggregation as claimed in claim 1, characterized in that: The key point pre-selection module uses BEV features Generate coarse instance features about vector points Obtaining instance maps through the segmentation head Where N represents the number of categories of pre-selected instances.

6. The vector map construction method based on view interactive aggregation as claimed in claim 5, characterized in that: Mapping from an instance The key vector points are selected and the importance of the key points is learned. The key points with high weights are used as the initial vector points, and their geometric positions are output as pre-selected vector points Pres xy ∈N×P×2, N represents the number of instance categories, and P represents the number of points in that category.

7. The vector map construction method based on view interactive aggregation as claimed in claim 6, characterized in that: By adding the depth instance feature Depth of the pre-selected vector point instfea Back Projection Rev proj The initial position feature map is then transferred to the original position feature map to obtain the corresponding depth instance feature, and the standardized pixel coordinates of the vector points in the instance map are encoded to obtain the geometric position feature.

8. The vector map construction method based on view interactive aggregation as claimed in claim 7, characterized in that: The obtained deep instance feature Depth instfea and geometric position feature Pos instfea , which is the feature of the final pre-selected key point, and is used as the initial reference point P key .

9. The vector map construction method based on view interactive aggregation as claimed in claim 1, characterized in that: Decode the result of each map instance as a point query Will Initialize a random variable as the instance object Query, and query Q with the selected key points key Query by Example I Perform the addition operation as the Query input Q of the target detection algorithm tot =Q key +Q I , embed BEV into F ins As the Key / Value input of the target detection algorithm. The initial reference point P key It is input into the target detection algorithm as an additional hint feature, and a linear layer combined with a normalization layer and an activation function is used as the task head to predict the label and location of the instance, that is, to obtain the predicted vector map element label and location.

10. A vector map construction system implementing the method according to any one of claims 1 to 9, characterized in that: include: The feature extraction module is configured to: acquire a visible light image, and extract features of the visible light image in a pixel-level coordinate system and a camera coordinate system respectively; The view fusion module is configured as follows: the features in the pixel-level coordinate system and the features in the camera coordinate system are fused by an asynchronous mutual learning fusion module; The feature decoding module is configured to: fuse the features through the key point pre-selection module to obtain the deep instance features and geometric position features, and use them as the initial reference points; The fused features, key point query and instance query obtained based on the key point pre-selection module are used as the input of the target detection algorithm. The obtained initial reference point is used as an additional prompt feature of the target detection algorithm to obtain the predicted vector map element label and its location. The feature enhancement module is configured to: form a single-frame vector map with the predicted vector map element labels and their locations, aggregate the rich feature information in the historical frames with the sparse features in the current frame to enhance the sparse features of the current frame, and further obtain the vector map through the detection head.

Images