Vector map construction method and system based on view interaction aggregation

Abstract

The application relates to a vector map construction method and system based on view interaction aggregation, visible light images are acquired, features of the visible light images under a pixel level coordinate system and a camera coordinate system are respectively extracted, and fused features are obtained through an asynchronous mutual learning fusion module; the fused features pass through a key point preselection module to obtain depth instance features and geometric position features, and the depth instance features and the geometric position features are used as initial reference points; the fused features, key point queries and instance queries obtained based on the key point preselection module are used as inputs of a target detection algorithm, the initial reference points are used as additional prompt features of the target detection algorithm, predicted vector map element labels and positions are obtained, single-frame vector maps are formed by the predicted vector map element labels and the positions, rich feature information in historical frames is aggregated with sparse features in a current frame to enhance the sparse features in the current frame, and a vector map is further obtained through a detection head.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of map construction, in particular to a vector map construction method and system based on view interaction aggregation. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute the prior art.

[0003] During the implementation of automatic driving of a vehicle, map data needs to be updated in real time, i.e., a map is constructed through an algorithm. Many methods of constructing a map excessively rely on depth information, and when the depth estimation is inaccurate, the quality of the generated map is greatly reduced. At the same time, due to the fact that the method based on vision is sensitive to extreme situations such as insufficient light and object occlusion, and environmental factors such as rain, fog and weak light, the information is sparse in the map construction, resulting in incomplete and inaccurate construction. Therefore, the existing technology begins to focus on using the BEV (Bird Eye’s View, bird’s eye view) technology to construct a vector map with higher precision, helping the positioning and planning of an autonomous vehicle.

[0004] The existing BEV map generation is mainly divided into two modes: map rasterization generation and map vectorization generation. The map rasterization method uses key feature points extracted from sensor data to construct a map. However, due to the difficulty in distinguishing different map elements and the inability to provide rich structural information, a large amount of post-processing work is required to make the map available for downstream tasks.

[0005] To solve the above problems, the map vectorization method has become the mainstream direction of research on real-time construction of high-precision maps. Compared with the map rasterization method, the map is generated in a vectorized manner, which has significant advantages in updating and maintaining map information elements. Not only can the map information in the driving process be restored, but also the integrity and real-time performance of the information can be ensured. However, the existing method has two significant problems:

[0006] First, there is an incomplete problem in considering local information and global information. The traditional feature fusion method will cause feature redundancy, especially when constructing a vector map, this fusion strategy may affect the distinguishability of the map instances.

[0007] Second, in the process of constructing a vector map, there is a problem of sparse visual information features. The existing instance feature extraction method has not been able to fully capture all relevant visual information, resulting in the problem of missing details in the map caused by information sparsity. SUMMARY

[0008] In order to solve the technical problems in the above background art, the application provides a vector map construction method and system based on view interaction aggregation, which can correct the geometric distortion of long-distance lane lines encountered by an autonomous vehicle during driving, optimize the feature sparsity problem caused by occlusion in a single frame scene, and further generate a more complete BEV vector map to improve the accuracy of positioning and decision-making in an autonomous driving system.

[0009] In order to achieve the above purpose, the application adopts the following technical solutions:

[0010] The first aspect of the application provides a vector map construction method based on view interaction aggregation, comprising the following steps:

[0011] Obtain a visible light image, and extract features of the visible light image in a pixel-level coordinate system and a camera coordinate system, respectively;

[0012] The features in the pixel-level coordinate system and the features in the camera coordinate system are fused through an asynchronous mutual learning fusion module to obtain fused features;

[0013] The fused features are input into a key point pre-selection module to obtain depth instance features and geometric position features, which are used as initial reference points; the fused features, key point queries and instance queries obtained based on the key point pre-selection module are used as inputs of a target detection algorithm, the initial reference points are used as additional hint features of the target detection algorithm, and predicted vector map element labels and their positions are obtained;

[0014] The predicted vector map element labels and their positions form a single-frame vector map, rich feature information in historical frames is aggregated with sparse features in a current frame to enhance the sparse features in the current frame, and a vector map is further obtained through a detection head.

[0015] Further, the asynchronous mutual learning fusion module learns the features F pixel decoded from the perspective view in the camera coordinate system F p from each other, specifically: in the first N rounds of training, the features extracted in the pixel-level coordinate system are selected as the teacher of the preliminary map features, and the spatial loss in the camera coordinate system is calculated for supervision; when the number of training reaches a set boundary value, the learned preliminary map features are used as the teacher of the features extracted in the pixel-level coordinate system, the foreground and background of the predicted feature map are used to realize mutual learning between the two, and fused features are obtained.

[0016] Further, the asynchronous mutual learning fusion module is as follows:

[0017]

[0018] Among them, LRDM represents the asynchronous mutual learning model, x1 in LRDM(x1,x2) represents the guided feature, x2 represents the feature to be learned, epoch is the number of training times, and F pixel is the feature extracted in the pixel-level coordinate system, F p It is the decoded perspective view feature in the camera coordinate system.

[0019] Furthermore, the key point pre-selection module selects initial vector points from the sparse instance map, obtains a point query by mixing the semantic features and position features of each vector point, and then performs multi-layer optimization in the decoder.

[0020] Furthermore, the key point pre-selection module uses BEV characteristics Generate coarse instance features about vector points Obtaining instance maps through the segmentation head Where N represents the number of categories of pre-selected instances.

[0021] Further, from the instance mapping The key vector points are selected and the importance of the key points is learned. The key points with high weights are used as initial vector points, and their geometric positions are output as pre-selected vector points Pres xy ∈N×P×2, N represents the number of instance categories, and P represents the number of points in that category.

[0022] Furthermore, by preselecting the depth instance feature of the vector point Depth instfea Back Projection Rev proj The initial position feature map is then passed to obtain the corresponding depth instance feature, and the standardized pixel coordinates of the vector points in the instance map are encoded to obtain the geometric position feature.

[0023] Furthermore, the obtained deep instance feature Depth instfea and geometric position feature Pos instfea , which is the feature of the final pre-selected key point, and is used as the initial reference point P key .

[0024] Furthermore, the result of decoding each map instance is used as a point query Will Initialize a random variable as the instance object Query, and query the selected key points Q key Query by Example Q I Perform the addition operation as the Query input Q of the target detection algorithm tot =Q key +Q I , embed BEV into Fins Key / Value input of the target detection algorithm. Initial reference point P key As an additional prompt feature input into the target detection algorithm, through a linear layer combined with a normalization layer and a RELU activation function as a task head, the label and the location of the predicted instance are obtained, that is, the predicted vector map element label and the location thereof.

[0025] The second aspect of the application provides a system required to implement the above method, comprising:

[0026] The data acquisition and preprocessing module is configured to:

[0027] The feature extraction module is configured to: acquire the visible light image, and extract features of the visible light image in a pixel-level coordinate system and a camera coordinate system, respectively;

[0028] The view fusion module is configured to: the features in the pixel-level coordinate system and the features in the camera coordinate system are fused into fused features through an asynchronous mutual learning fusion module;

[0029] The feature decoding module is configured to: the fused features are input into the key point preselection module to obtain depth instance features and geometric position features, and the depth instance features and the geometric position features are used as initial reference points; the fused features, the key points obtained based on the key point preselection module, and instance queries are used as inputs of a target detection algorithm, and the initial reference points are used as additional prompt features of the target detection algorithm, so that a predicted vector map element label and a location thereof are obtained;

[0030] The feature enhancement module is configured to: the predicted vector map element label and the location thereof form a single-frame vector map, rich feature information in historical frames is aggregated with sparse features in a current frame to enhance the sparse features in the current frame, and a vector map is further obtained through a detection head.

[0031] The third aspect of the application provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the steps in the above vector map construction based on view interaction aggregation.

[0032] The fourth aspect of the application provides a computer device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps in the above vector map construction based on view interaction aggregation when executing the program.

[0033] The fifth aspect of the application provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the steps in the above vector map construction based on view interaction aggregation.

[0034] Compared with the prior art, the above one or more technical solutions have the following beneficial effects:

[0035] 1. Since one of the main methods to realize BEV semantic mapping is to design a Transformer for deep learning of images, this Transformer can display the BEV mapping of depth estimation. When designing a view Transformer for deep learning in the pixel coordinate system, the early learning speed is slow, but once the training is completed, the effect is significant. In addition, another mainstream method to realize BEV semantic mapping is to display depth estimation through a perspective view in the camera coordinate system for BEV mapping with depth estimation, and assign a hypothetical depth to each pixel. However, due to the near-far feature of the perspective effect, the feature extraction of the same object in the image is distorted. Therefore, by using the asynchronous mutual learning fusion module, the advantages of the pixel coordinate system and the camera coordinate system are combined to optimize the learning process of the BEV semantic mapping, and a complete lane line representation under the bird's eye view is obtained.

[0036] 2. A key point pre-selection module is designed for the region of interest for feature decoding to pay more attention to the vectorized features of the target region during decoding, thereby improving the accuracy of instance prediction by pre-selecting the unclassified features.

[0037] 3. Through instance feature aggregation enhancement, the challenge of sparse target lane line features caused by occlusion in a single frame scene is addressed, and the correlation of sparse features in adjacent frames in the time sequence is aggregated. By using the relatively rich features extracted in the previous frame to enhance the expression of sparse features in the current frame, the sparse instance features of the current frame can be made more abundant. BRIEF DESCRIPTION OF DRAWINGS

[0038] The drawings accompanying the specification of this application form a part thereof, serve to provide further understanding of the application, and together with the description, explain the application. The specific embodiments of the application and their description are used to explain the application, and do not constitute an improper limitation of the application.

[0039] Figure 1 is a whole flowchart of the BEV vectorization map construction process based on view interaction aggregation in an autonomous driving scene provided by one or more embodiments of the application;

[0040] Figure 2 is a structure schematic diagram of the asynchronous mutual learning fusion module provided by one or more embodiments of the application;

[0041] Figure 3 is a structure schematic diagram of the key point pre-selection module provided by one or more embodiments of the application;

[0042] Figure 4is an example feature aggregation enhancement module structure schematic diagram provided by one or more embodiments of the present application;

[0043] Figure 5 is a BEV vectorization map construction system based on view interaction aggregation in an autonomous driving scenario provided by one or more embodiments of the present application;

[0044] Figure 6 is an electronic device structure schematic diagram provided by one or more embodiments of the present application. DETAILED DESCRIPTION

[0045] The present application will be further described below in conjunction with the accompanying drawings and embodiments.

[0046] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present application. Unless otherwise indicated, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs.

[0047] Embodiment one:

[0048] The vector map construction method based on view interaction aggregation includes the following steps:

[0049] Real-time acquisition of visible light images of a map region to be generated;

[0050] Based on the obtained visible light images, a feature extraction module is used to extract features of the images in a pixel-level coordinate system and a camera coordinate system, and then the data in the camera coordinate system is preliminarily decoded to obtain preliminary map features.

[0051] An asynchronous mutual learning fusion module (Long-range distortion rectification Module, LDRM) is used to mutually learn the features extracted in the pixel-level coordinate system and the decoded perspective view features in the camera coordinate system. Specifically:

[0052] In the first N rounds of training, the features extracted in the pixel-level coordinate system have certain prior knowledge and stronger learning ability, and can transfer the learned knowledge to the features with relatively weak ability, so the features extracted in the pixel-level coordinate system are selected as the teacher of the preliminary map features, and the spatial loss in the camera coordinate system is calculated for supervision. When the epoch reaches the set boundary value, the learned preliminary map features are used as the teacher of the features extracted in the pixel-level coordinate system, and the mutual learning of the foreground and background of the feature map is realized through prediction to finally achieve the effect of view fusion.

[0053] In the feature decoding module, the region of interest of the Deformable DETR in the traditional mode is greatly limited by the input features. By using a key point preselection module (KPPM), initial vector points are selected from sparse instance mapping, and the semantic features and position features of each vector point are mixed to obtain a point query, which is then optimized in the decoder through multiple layers.

[0054] By using an instance feature aggregation enhancement module (IFAEM), based on the instance features in the historical frames and the current frame, with the help of the instance box, label and instance score under supervision, a consistency learning strategy is formulated to aggregate the rich feature information in the historical frames and the sparse features in the current frame to enhance the sparse features of the current frame. Considering that different labels have unique clustering spaces, positive and negative samples are selected independently under different labels to ensure the universality of the contrast learning.

[0055] As shown in Figure 1 , the embodiment takes the nuScenes dataset and the Argoverse2 dataset as examples to provide a BEV vectorization map construction method based on view interaction aggregation in an autonomous driving scenario, which specifically includes the following processing procedures:

[0056] Step one, data loading: real-time acquisition of RGB image data of the map area to be generated as model input.

[0057] In specific implementation, based on the vehicle-mounted sensor, visible light images of the map area to be generated are collected in real time, wherein the vehicle-mounted sensor is integrated with a camera device and a laser radar device.

[0058] Step two, feature extraction: the input camera data is respectively subjected to feature extraction in the pixel coordinate system and the camera coordinate system, and the extracted features in the camera coordinate system are subjected to preliminary encoding and decoding to obtain results as input for subsequent feature fusion.

[0059] In specific implementation, the RGB image collected by the camera is taken as input, , represents the RGB picture input, , represents a three-dimensional matrix; W is the width; H is the width; C represents the number of channels. The feature extraction F pixel in the pixel coordinate system and the feature extraction F camera in the camera coordinate system are respectively performed.

[0060]

[0061] wherein θ i (·) represents the full connection layer of the i-th layer.

[0062] Step three, view fusion: the extracted pixel features and the preliminary results in the camera coordinate system are fused by the asynchronous mutual learning module (LDRM) to obtain the depth information and the fused results of the preliminary decoding in the perspective view.

[0063] Specifically, as shown in the figure, Figure 2 the asynchronous mutual learning module (LRDM) learns the two input features interactively to fuse the features extracted in different coordinate systems.

[0064] Asynchronous mutual learning module LRDM: the features F pixel extracted in the pixel-level coordinate system and the decoded perspective view features F p in the camera coordinate system are learned interactively. First, in the first N epochs of training, the features in the pixel-level coordinate system have certain prior knowledge and stronger learning ability, and can transfer the learned knowledge to the features with relatively weak ability, so F pixel is selected as the teacher of F p , and the spatial loss in the camera coordinate system is calculated for supervision. When the epoch reaches the set boundary value, F p is selected as the teacher of F pixel , and the mutual learning of the foreground and background of the predicted feature map is realized to finally achieve the effect of view fusion. Wherein LRDM represents the asynchronous mutual learning model, LRDM(x1, x2) wherein x1 represents the guided feature, and x2 represents the feature to be learned.

[0065]

[0066] Step four, feature decoding: the BEV features are decoded using the DETR framework to obtain the map elements of the vectorized map. Wherein the query selection of the key points adopts the key point pre-selection module (KPPM) of the embodiment.

[0067] Specifically, as shown in the figure, Figure 3 the key point pre-selection module pre-selects the unclassified features to further improve the accuracy of instance prediction.

[0068] Key point pre-selection module KPPM: a deformable DETR is used for decoding, wherein the input of the decoder includes a fused feature obtained from the asynchronous mutual learning module as the BEV embedding F ins , an instance query (wherein N represents the number of instance categories, and C represents the channel number of the feature), and an initial reference point (N p Query

[0069] In this embodiment, the BEV embedding is the BEV feature extracted by the encoder, i.e., the fusion feature obtained in step three. Since this feature needs to be used as an embedding to assist the subsequent Transformer in understanding the Query, this embodiment names it BEV embedding.

[0070] For the selection of key point queries, first use the BEV characteristics Generate coarse instance features about vector points Use a simple segmentation head to obtain instance mapping Where N represents the number of pre-selected instance categories. Then the instance mapping can be represented in the form mapped by the network:

[0071]

[0072]

[0073] Where is a complete three-layer ResNet structure, is a segmentation head structure.

[0074] For the instance mapping segmented out, use the key point selection method to select the key vector points, and propose a weight module to learn the importance of the key points. The key points with high weights are used as initial vector points, and their geometric positions are output as pre-selected vector points Pres xy ∈N×P×2(N represents the number of instance categories, and P represents the number of points in the category number):

[0075] Weight map =W(Map ins );

[0076] Pres xy =Norm(TopK(Flatten(Weight map ));

[0077] Where W(·) represents the weight module to be learned, TopK is used to select the highest position and output the position coordinates, and Norm is used here to realize the normalization of the coordinates to avoid the inconsistency of the coordinate system. By projecting the depth instance features Depth instfea Rev proj of these points back to the initial position feature map, the depth instance features of these points are obtained:

[0078] Pres′ xy =Pres xy [h, w] T ;

[0079] Pres xy =Pres′ xy [0]·Pres′ xy [1]*w;

[0080] Depth instfea =Rev proj (Flatten(F ins ), Pres″ xy );

[0081] Due to Depth instfea Due to the lack of relative geometric features, two MLP layers M2 are used to encode the normalized pixel coordinates of vector points in the instance map. The geometric position features are represented as follows:

[0082] Pos instfea =M2(Pres xy )

[0083] The final pre-selected key point features are composed of the depth instance features Depth instfea and geometric position feature Pos instfea constituted, which is used as the initial reference point P key :

[0084] P key =Depth instfea +Pos instfea

[0085] At the same time, the result of decoding each map instance is used as a point query Will Initialize a random variable as the instance object Query, and finally query the selected key point Q key Query by Example Q I Perform the addition operation as the query input Q of Deformable DET R tot =Q key +Q I , embed BEV into F ins As the Key / Value input of Deformable DETR. Initial reference point P key As additional hint features, they are input into Deformable DET R to help instance recognition and prediction. Finally, a linear layer combined with a normalization layer and a RELU activation function is used as the task head to predict the instance label and location.

[0086] Instances, or vector map elements, refer to specific, independent map elements or objects. These instances have clear definitions and properties in the vector map and are used to describe key information such as roads, lanes, traffic signs, and obstacles.

[0087] The Deformable DETR in this embodiment is an object detection algorithm based on Transformer and deformable convolution.

[0088] Step 5: Feature enhancement: Due to the sparse features in single-frame occlusion scenes, weak features exist after decoding. These weak features are fed into the designed feature aggregation module (IFAEM) for correlation aggregation to generate enhanced instance features to improve the quality of the vectorized map.

[0089] Specifically, such as Figure 4 As shown, the instance feature aggregation module aggregates and enhances the sparse features in the current frame and the features in the historical frames to generate enhanced instance features to improve the quality of the vectorized map.

[0090] Instance Feature Aggregation Enhancement Module (IFAEM): Since the instance features aggregated from KPPM are the output of single-frame perceptual information, and instance features extracted from occluded scenes are very sparse, the single-frame instance features processed by KPPM serve as the input to IFAEM. Based on the instance features from previous and current frames, the IFAEM module utilizes instance boxes, labels, and supervised instance scores to develop a consistent learning strategy. This module aggregates the rich feature information from previous frames with the sparse features from the current frame to enhance the sparse features of the current frame. Considering the unique clustering spaces between different labels, positive and negative samples are selected independently for each label to ensure the comprehensiveness of contrastive learning. The aggregated features with the highest instance scores for each semantic label are sent to the contrastive head. This score is generated by comprehensively evaluating instance and location accuracy under the supervision of ground truth (GT), resulting in the "v" embedding for contrastive learning. Since the number of queries is greater than the number of instances in each frame, the number "v" is designed to be a dynamic norm. A fixed maximum "v" is set for each label.

[0091] The selection mechanism of positive and negative samples is as follows: in the adjacent frames of the time sequence, the change of the dynamic environment may cause the feature difference between the instances with similar positions. Especially, due to occlusion, part of the instance features are sparse. However, through the learning process of positive samples, the sparse feature expression can be enhanced from relatively rich features. Therefore, the instance features in the historical frames within a certain range around the "v" instance are selected inst As a positive sample sample T . Negative sample sample F is selected from the negative label, which allows the features of instances with the same label to be pulled closer together. The negative label refers to a category other than "v". The number of negative samples for each negative label is fixed. The instance sample with the highest score in the current frame is selected as the negative sample to improve learning efficiency.

[0092]

[0093]

[0094]

[0095] where Label T and Label F are the selected labels, and k is a fixed value.

[0096] Step six: send the features output by the enhancement module into the detection head to obtain the final detection result.

[0097] In the current scheme, considering the negative effects of image geometric distortion and the positive and negative sample errors in instance consistency learning, which may lead to prediction errors when the application is applied in actual map construction tasks, the application designs a global loss function, which realizes the consideration of the negative effects of both through the back propagation of the model, as follows:

[0098] Considering that geometric distortion will be brought into the fusion process of features in the pixel coordinate system and the decoded results in the camera coordinate system if not checked, the difference between the decoded perspective view features in the camera coordinate system and the ground truth is calculated using cross-entropy loss, to ensure the accuracy of the generated BEV route map and reduce the influence of any geometric distortion that may occur:

[0099]

[0100] where y represents the number of cameras, L xy is the cross-entropy loss.

[0101] Considering that there are errors in positive and negative samples in instance consistency learning, since the selection of positive and negative samples can be regarded as training a classification model, the performance of the model in correctly selecting positive and negative samples is improved by minimizing the difference between the predicted probability distribution of the model and the real label probability distribution:

[0102] Loss IFAEM =log[1+∑ cls ∑ Sam +∑ Sam -exp(v cls *Sam - -v cls *Sam + )];

[0103] wherein v cls represents the dynamic norm of each class, Sam + and Sam - represent positive and negative samples.

[0104] Therefore, a global loss function Loss is proposed for the embodiment to solve the geometric distortion and positive and negative sample errors of instance consistency learning, so as to improve the accuracy of map prediction generation. The specific representation is as follows:

[0105] Loss=Loss LRDM +Loss IFAEM .

[0106] Since one of the main methods to realize BEV semantic mapping is to design a Transformer for deep learning of images, which can display the BEV mapping of depth estimation. When designing a view Transformer for deep learning in the pixel coordinate system, a challenge is faced: the learning speed is slow in the early stage, but once the training is completed, the effect is significant. Another mainstream method to realize BEV semantic mapping is to display depth estimation through a perspective view in the camera coordinate system for BEV mapping with depth estimation, and assign a hypothetical depth to each pixel. However, this method also brings some problems, such as the near-large and far-small characteristics caused by the perspective effect, which makes the feature extraction of the same object in the image distorted.

[0107] Therefore, the embodiment gives a BEV vectorization map construction method based on view interaction aggregation in an autonomous driving scene, and proposes an asynchronous mutual learning fusion module, aiming to optimize the learning process of BEV semantic mapping by combining the advantages of the pixel coordinate system and the camera coordinate system, and obtain a complete lane line representation in the bird's eye view perspective.

[0108] A key point pre-selection module is designed for the region of interest of feature decoding to pay more attention to the vectorized features of the target region during decoding. Currently, in the process of making a vectorized map, the generation of map element instances depends on the DETR architecture, which can decode these instances from the BEV features. These initial queries are randomly generated, and all sampling points use uniform learning dimensions. After a large amount of data training, the greatest common factor in the training set is selected as the initial value of the query. However, for those vector point instances far from the center point, the regression learning process needs more iteration times to converge. This makes the current initialization method inefficient when dealing with high-definition maps containing multiple shape instances. Therefore, in order to improve the efficiency and make the target region be paid more attention to, the embodiment designs a key point pre-selection module, which can select key points differently according to different instance features, thereby more effectively aggregating instance features, and the module can be embedded into any map generation algorithm with Deformable DETR as the decoder, having good generality.

[0109] A new component of instance feature aggregation enhancement is also designed to deal with the challenge of sparse target lane line features caused by occlusion in single-frame scenarios. It can aggregate the sparse features in adjacent frames in the time sequence, enhance the expression of sparse features in the current frame by using the more abundant features extracted in the previous frame, so that the sparse instance features of the current frame can be more abundant.

[0110] The method in the above embodiment can be directly embodied by a hardware processor to complete, or be completed by a combination of hardware and software modules in the processor. The software module can be located in a mature storage medium in the field, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and combines the hardware to complete the steps of the above method. To avoid repetition, it will not be described in detail here.

[0111] Embodiment two:

[0112] As shown in Figure 5 , a system for implementing the above method includes:

[0113] The feature extraction module is configured to obtain a visible light image and extract features of the visible light image in a pixel-level coordinate system and a camera coordinate system, respectively;

[0114] The view fusion module is configured to obtain fused features by asynchronous mutual learning feature fusion of the features in the pixel-level coordinate system and the features in the camera coordinate system;

[0115] The feature decoding module is configured to: fuse the features to obtain deep instance features and geometric position features through the key point preselection module, and take the deep instance features and the geometric position features as initial reference points; take the fused features and the key points obtained based on the key point preselection module as inputs of a target detection algorithm, take the initial reference points as additional hint features of the target detection algorithm, and obtain predicted vector map element labels and positions thereof; key and the instance query Q I are taken as inputs of a target detection algorithm, the initial reference points obtained are taken as additional hint features of the target detection algorithm, and predicted vector map element labels and positions thereof are obtained.

[0116] The feature enhancement module is configured to: form a single-frame vector map based on the predicted vector map element labels and the positions thereof, aggregate rich feature information in historical frames with sparse features in a current frame to enhance the sparse features in the current frame, and further obtain the vector map through the detection head.

[0117] It can be understood that the system is consistent with the method in the above-mentioned embodiments, and the specific details are described in detail in the method embodiments, so the details are not repeated.

[0118] Embodiment three:

[0119] The embodiment provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to realize the steps in the vector map construction based on view interaction aggregation in the above-mentioned embodiment one.

[0120] In the embodiment, as shown in the figure, Figure 6 The design of the processor should be considered flexible and diverse to adapt to different application requirements. Specifically, the processor can be configured to include one or more cores, such as a single-core processor, a four-core processor, an eight-core processor, etc., to provide different levels of computing power. In terms of hardware implementation, the processor can be based on a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, as well as discrete gate or transistor logic devices and discrete hardware components. The choice of these hardware forms depends on specific performance requirements and cost-effectiveness considerations.

[0121] Further, the processor architecture can include a combination of a main processor and a co-processor. The main processor, often referred to as the central processing unit (CPU), is responsible for processing the computing tasks of the device in the wake-up state. The co-processor is a low-power processor designed specifically for processing data in standby mode to optimize energy consumption and extend the service life of the device. In some embodiments, the processor can be integrated with a graphics processing unit (GPU), which is responsible for rendering and drawing the content required for the display screen. This integrated design helps to improve the efficiency of graphics processing and may reduce the overall power consumption of the system.

[0122] Embodiment four:

[0123] The embodiment provides a computer device, including a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the steps in the vector map construction based on view interaction aggregation of the above-embodiment one when executing the program.

[0124] In the embodiment, the design of the memory aims to provide an efficient and reliable data storage solution. The memory system is composed of one or more computer-readable storage media, which should have non-transient characteristics to ensure the persistence of data. Specifically, the memory configuration includes high-speed random access memory (RAM) and non-volatile memory (NVM), which can be further subdivided into disk storage devices and flash storage devices such as solid-state drives (SSD) and USB flash drives. The main function of the memory is to store computer program codes, which, when loaded and executed by the processor, can implement the relevant steps of the method executed by the electronic device side disclosed in any of the preceding embodiments. In addition, the memory is also responsible for storing operating systems, application programs and data resources, which can be temporary storage or permanent storage. The choice of storage method depends on the frequency of data use and access requirements. The choice of operating system should cover mainstream options, including but not limited to Windows, Linux and Android, to meet the needs of different users and application scenarios. Such a design not only provides flexibility, but also ensures compatibility with existing software ecosystems, thereby enhancing the applicability and user-friendliness of the electronic device. Through such a comprehensive storage solution, the embodiment aims to achieve efficient management of data while maintaining the stability and response speed of the system.

[0125] Embodiment five:

[0126] The embodiment provides a computer program product, including a computer program, and the computer program, when executed by a processor, implements the steps in the vector map construction based on view interaction aggregation of the above-embodiment one.

[0127] The steps involved in the above embodiments two to five correspond to embodiment one, and the specific implementation can refer to the relevant description part of embodiment one. The term "computer-readable storage medium" should be understood to include a single medium or multiple media of one or more instruction sets; it should also be understood to include any medium that can store, encode or carry instruction sets for execution by a processor and make the processor execute any of the methods in the present application.

[0128] The above merely provides the preferred embodiments of the present application, and is not used to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modifications, equivalent replacements, improvements, etc. made within the principles and technical scope of the present application shall fall into the scope of the present application.

Claims

1. A vector map construction method based on view interaction aggregation, characterized by: The following steps are involved: Obtain a visible light image and extract features of the visible light image in the pixel-level coordinate system and the camera coordinate system respectively; The features in the pixel-level coordinate system and the features in the camera coordinate system are fused through an asynchronous mutual learning fusion module to obtain fused features; The fusion features are passed through the key point pre-selection module to obtain the deep instance features and geometric position features, which are used as the initial reference points; The fused features, along with key point queries and instance queries obtained from the key point pre-selection module, are used as input to the object detection algorithm. The obtained initial reference points serve as additional prompt features for the object detection algorithm to obtain the predicted vector map element labels and locations. The predicted vector map element labels and locations form a single-frame vector map. The rich feature information in the historical frames is aggregated with the sparse features in the current frame to enhance the sparse features of the current frame, and the vector map is further obtained through the detection head. Among them, the asynchronous mutual learning fusion module extracts the features in the pixel-level coordinate system Perspective view features decoded from camera coordinate system Mutual learning is carried out. Specifically, in the first N rounds of training, the features extracted in the pixel-level coordinate system are selected as the teacher of the preliminary map features, and the spatial loss is calculated in the camera coordinate system for supervision. When the number of training times reaches the set boundary value, the learned preliminary map features are used as the teacher of the features extracted in the pixel-level coordinate system. By predicting the foreground and background of the feature map, the two achieve mutual learning and obtain fused features. The key point pre-selection module uses BEV features Generate coarse instance features about vector points , instance mapping is obtained by segmentation head , where N represents the number of categories of pre-selected instances; Represents a three-dimensional matrix; is the width; is the width; Indicates the number of channels; The result of decoding each map instance is used as a point query Will Initialize a random variable as the instance object Query, and query the selected key points Query by Example Perform the addition operation as the query input of the target detection algorithm , embedding BEV into As the Key / Value input of the target detection algorithm; initial reference point It is input into the target detection algorithm as an additional hint feature, and a linear layer combined with a normalization layer and an activation function is used as the task head to predict the instance label and location, that is, to obtain the predicted vector map element label and location.

2. The vector map construction method based on view interactive aggregation as claimed in claim 1, characterized in that: The asynchronous mutual learning fusion module is shown in the following formula: ； in, represents the asynchronous mutual learning model, middle Indicates the characteristics of guidance, represents the features to be learned, is the number of training times, is the feature extracted in the pixel-level coordinate system, It is the decoded perspective view feature in the camera coordinate system.

3. The vector map construction method based on view interactive aggregation as claimed in claim 1, characterized in that: The key point pre-selection module selects the initial vector points from the sparse instance map, obtains a point query by mixing the semantic features and position features of each vector point, and then performs multi-layer optimization in the decoder.

4. The vector map construction method based on view interactive aggregation as claimed in claim 1, characterized in that: Mapping from an instance Select key vector points and learn the importance of key points. Key points with high weights are used as initial vector points, and their geometric positions are output as pre-selected vector points. , N represents the number of instance categories, and P represents the number of points in that category.

5. The vector map construction method based on view interactive aggregation as claimed in claim 4, characterized in that: By adding the depth instance features of the preselected vector points Back projection The initial position feature map is then passed to obtain the corresponding depth instance feature, and the standardized pixel coordinates of the vector points in the instance map are encoded to obtain the geometric position feature.

6. The vector map construction method based on view interactive aggregation as claimed in claim 5, characterized in that: The obtained deep instance features and geometric position features , which is the feature of the final pre-selected key point, and is used as the initial reference point .

7. A vector map construction system implementing the method according to any one of claims 1 to 6, characterized in that: include: The feature extraction module is configured to: acquire a visible light image and extract features of the visible light image in a pixel-level coordinate system and a camera coordinate system respectively; The view fusion module is configured to: combine features in the pixel-level coordinate system and features in the camera coordinate system, and obtain fused features through an asynchronous mutual learning fusion module; The feature decoding module is configured to: fuse the features through the key point pre-selection module to obtain the deep instance features and geometric position features, which are used as the initial reference points; The fused features, along with key point queries and instance queries obtained from the key point pre-selection module, are used as input to the object detection algorithm. The obtained initial reference points serve as additional prompt features for the object detection algorithm to obtain the predicted vector map element labels and locations. The feature enhancement module is configured to: form a single-frame vector map with the predicted vector map element labels and their locations, aggregate the rich feature information in the historical frames with the sparse features in the current frame to enhance the sparse features of the current frame, and further obtain a vector map through the detection head.

Images