Three-dimensional target detection method and device based on map token guidance

Abstract

The application discloses a three-dimensional target detection method and device based on map token guidance, which comprises the following steps: encoding sensor data to obtain an aerial view feature; obtaining a high-precision map vector polyline set corresponding to a scene to be detected; encoding each polyline to generate a map token that can be directly read by a detection query, and obtaining a map token set; extracting road elements from the high-precision map to generate a road mask on the aerial view; performing dynamic sampling based on the road mask and mixing with uniform prior query points to obtain final detection query points as starting reference points of the detection query; and feeding the aerial view feature into a detection decoder to perform map token cross-attention operation on the query at each layer or part of the key layer, so that the query reads map information in the iteration process, and finally outputs a three-dimensional target detection result. According to the application, the query obtains map prior guidance from the starting position to the whole iteration updating process, and the three-dimensional target detection precision in a complex road scene is effectively improved.

Description

Technical Field

[0001] This invention belongs to the field of autonomous driving environmental perception and 3D target detection technology, specifically relating to a 3D target detection method and device based on map token guidance. Background Technology

[0002] In the field of autonomous driving, 3D target detection methods based on the Transformer query mechanism are an important technical approach for environmental perception. These methods typically first construct a bird's-eye view feature map, then iteratively compute a set of query vectors through a multi-layer decoder to output the detection results, including the position, size, and orientation of the 3D target.

[0003] Existing query-based 3D object detection methods mostly use uniformly distributed learnable reference points for query initialization. Even when high-precision map information is incorporated into the detection process, it is mostly used only for bird's-eye view feature fusion, region of interest cropping, or post-detection result constraints, and it is difficult to directly intervene in the source of query initialization or the process of each iteration update of the query. In complex road scenarios such as complex intersections, curves, and multi-lane roads, relying solely on visual bird's-eye view convolution is insufficient to obtain clear road structure priors for long-distance queries.

[0004] Existing 3D target detection technologies that integrate high-precision maps can be mainly divided into several categories: The first type utilizes high-precision maps to generate drivable area masks, projects point clouds onto these masks, extracts point clouds within the drivable area, and then performs 3D target detection. This type of solution involves data preprocessing before detection; map information cannot be used for the detection query itself.

[0005] The second category mainly focuses on the fusion detection of camera images and LiDAR point clouds in a bird's-eye view space, and uses information such as the fused heatmap and point cloud feature vectors to initialize query vectors. The initialization basis of this type of scheme mainly comes from multimodal perception features, rather than the road structure in high-precision maps, and map information is not used for query generation and guidance.

[0006] The third type utilizes map information to prioritize sensor processing in autonomous vehicles, such as changing the camera's field of view or focusing on key areas based on road features. This approach uses maps as an external priority reference but does not encode map vector objects into interactive units that can be directly read by detection queries, nor does it involve layer-by-layer map guidance for decoder queries.

[0007] In summary, existing 3D target detection methods cannot directly utilize map information for query generation and update, leading to a disconnect between query initialization and road geometry. Long-distance queries struggle to obtain a clear road network structure, and detection accuracy is limited in complex road scenarios. Summary of the Invention

[0008] To address the aforementioned technical problems, this invention proposes a three-dimensional target detection method and apparatus based on map token guidance.

[0009] To achieve the above objectives, the technical solution of the present invention is as follows: In a first aspect, this invention discloses a three-dimensional target detection method based on map token guidance, comprising: Step S1: Encode the collected sensor data into bird's-eye view features; Step S2: Obtain the set of high-precision map vector polylines corresponding to the scene to be detected. Each polyline in the vector polyline set includes: several valid points and corresponding polyline category labels; Step S3: Encode each polyline in the vector polyline set to generate a map token that can be directly read by the query, and obtain a map token set; Step S4: Extract road-related map features from the high-precision map and generate road surface masks on the bird's-eye view; Step S5: Perform dynamic sampling based on the road surface mask to generate road sampling query points, and then mix the road sampling query points with uniform prior query points to obtain the final detection query points; Step S6: Feed the bird's-eye view features into the detection decoder, using the final detection query point as the starting reference point for the detection query. Perform map token cross-attention operation on the query at each layer or some key layers of the detection decoder, so that the query reads map information in the map token set during the iteration process. After iteration by the detection decoder, output the 3D target detection result.

[0010] Based on the above technical solution, the following improvements can be made: As a preferred embodiment, step S3 includes: Step S3.1: Encode each valid point in the polyline to obtain a geometric description at the polyline level; Step S3.2: Perform category embedding on the polyline category labels to obtain the map semantic type; Step S3.3: Encode the position of the starting valid point of the polyline to obtain spatial position information; Step S3.4: Concatenate and map the geometric description, map semantic type, and spatial location information to the target dimension to obtain a map token consistent with the detection query dimension; Step S3.5: Traverse each polyline in the vector polyline set, repeating steps S3.1 to S3.4 to obtain the map token set.

[0011] As a preferred embodiment, before performing map token encoding on each polyline in the vector polyline set in step S3, the following steps are also included: The number of polylines in the vector polyline set is filled to a fixed number, and the number of valid points in each polyline is filled to a fixed number. Any insufficient points are padded with invalid values ​​to generate a validity mask. The validity mask is used to mark whether each polyline is a real and valid map feature.

[0012] As a preferred option, step S5, dynamic sampling based on the road surface mask, includes: The positions in the road surface mask below the threshold are set to zero, and the remaining positions are normalized to obtain the sampleable probability distribution; According to the preset number of queries, a corresponding number of bird's-eye view grid points are sampled from the probability distribution as the initial road sampling query points; Random perturbations are added to the coordinates of each initial road sampling query point to obtain the final road sampling query point.

[0013] As a preferred option, the mixing method of road sampling query points and uniform prior query points can be any one of linear mixing, splicing, grouping, gating selection, or gradually adjusting the mixing ratio according to the training stage.

[0014] As a preferred approach, in step S6, a map token cross-attention operation is performed on the query at each or some key layers of the detection decoder, satisfying: in, This is the query vector output by the i-th layer query decoder; Let be the update query vector output by the i-th layer decoder after being guided by map token cross attention; This is a cross-attention function used to calculate the correlation between the query vector and each token in the map token set, and to aggregate map information accordingly. A set of map tokens, serving as the key and value for the cross-attention function; This is a validity mask used to indicate whether each polyline is a valid map feature. Inject coefficients into the map residuals.

[0015] As a preferred approach, the map attention parameters can be shared between different layers of the detector, or set independently for each layer.

[0016] Secondly, this invention discloses a map token-guided 3D target detection device, comprising: The bird's-eye view encoding module is used to encode the collected sensor data into bird's-eye view features. The map acquisition module is used to acquire a set of high-precision map vector polylines corresponding to the scene to be detected. Each polyline in the vector polyline set includes: several valid points and a corresponding polyline category label; The map token encoding module is used to encode each polyline in the vector polyline set, generate map tokens that can be directly read by queries, and obtain a map token set; The road surface mask generation module is used to extract road-related map features from high-precision maps and generate road surface masks on bird's-eye views. The query point generation module is used to perform dynamic sampling based on the road surface mask to generate road sampling query points, and then mix the road sampling query points with uniform prior query points to obtain the final detection query points. The detection decoding module is used to feed the bird's-eye view features into the detection decoder, using the final detection query point as the starting reference point for the detection query. At each layer or some key layers of the detection decoder, map token cross-attention operation is performed on the query, so that the query reads map information in the map token set during the iteration process. After the detection decoder iterates, the 3D target detection result is output.

[0017] Thirdly, the present invention discloses a computing device, comprising: One or more processors; Memory; And one or more programs, wherein the one or more programs are stored in memory and configured to be executed by one or more processors, and the one or more programs include instructions for any of the map token-guided 3D object detection methods described above.

[0018] Fourthly, the present invention discloses a storage medium storing one or more computer-readable programs, the one or more programs including instructions adapted to be loaded by a memory and executed any of the above-described map token-guided 3D target detection methods.

[0019] This invention discloses a three-dimensional target detection method and apparatus based on map token guidance, which has the following beneficial effects: First, this invention encodes high-precision map vector polylines into map tokens that can be directly read by queries, so that map information no longer stays only at the network input feature layer, but enters the query inference layer and participates in the target detection process, fundamentally solving the technical problem that map information is difficult to directly participate in query generation.

[0020] Second, this invention performs map token cross-attention operations on the query at each or some key layers of the detection decoder, so that the query continuously obtains road geometry and semantic constraints during the iterative update process, rather than using the map only once at the input stage, effectively solving the problem that it is difficult to obtain a clear road network structure for long-distance queries.

[0021] Third, the present invention generates road sampling query points based on road surface masking through dynamic sampling, which enables the query to be preferentially concentrated on road-related areas from the initialization stage, thereby improving the utilization rate of effective queries and improving the defect of query initialization being disconnected from road geometry.

[0022] Fourth, this invention mixes road sampling query points with uniform prior query points, which not only uses map priors to guide queries to road areas, but also retains global search capabilities, avoiding the problem of insufficient global coverage caused by excessive concentration of queries in road areas.

[0023] Fifth, this method does not require rewriting or modifying the overall structure of the basic detection decoder; it can be implemented simply by adding map token encoding, sampling, and map token cross-attention related algorithms, thus possessing good engineering compatibility. Attached Figure Description

[0024] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 A flowchart of a three-dimensional target detection method provided in an embodiment of the present invention.

[0026] Figure 2 This is a schematic diagram of map token encoding provided in an embodiment of the present invention.

[0027] Figure 3 This is a schematic diagram of dynamic sampling of road surface mask provided in an embodiment of the present invention.

[0028] Figure 4 This is a schematic diagram illustrating the joint guidance of query initialization and iterative update provided in an embodiment of the present invention. Detailed Implementation

[0029] The preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0030] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0031] Using ordinal numbers such as “first,” “second,” “third,” etc. to describe ordinary objects merely indicates different instances of similar objects and is not intended to imply that the objects being described must have a given order in time, space, sequence, or any other way.

[0032] Furthermore, the expression "includes" is an "open-ended" expression, which means only that there is a corresponding component or step, and should not be interpreted as excluding additional components or steps.

[0033] In some embodiments, the present invention discloses a 3D target detection method based on map token guidance, such as... Figure 1 As shown, it includes: Step S1: Encode the collected sensor data into bird's-eye view features; Step S2: Obtain the set of high-precision map vector polylines corresponding to the scene to be detected. Each polyline in the vector polyline set includes: several valid points and corresponding polyline category labels; Step S3: Encode each polyline in the vector polyline set to generate a map token that can be directly read by the query, and obtain a map token set; Step S4: Extract road-related map features from the high-precision map and generate road surface masks on the bird's-eye view; Step S5: Perform dynamic sampling based on the road surface mask to generate road sampling query points, and then mix the road sampling query points with uniform prior query points to obtain the final detection query points; Step S6: Feed the bird's-eye view features into the detection decoder, using the final detection query point as the starting reference point for the detection query. Perform map token cross-attention operation on the query at each layer or some key layers of the detection decoder, so that the query reads map information in the map token set during the iteration process. After iteration by the detection decoder, output the 3D target detection result.

[0034] To facilitate understanding of this invention, it can be understood as adding a "map-reading guide" to the detection decoder of autonomous driving. On one hand, based on a high-precision map, some starting reference points for queries are placed near roads; on the other hand, map elements such as lane lines, road boundaries, and zebra crossings are encoded into several map tokens, allowing queries in the detection decoder to view these map tokens in each iteration. In this way, queries in the detection decoder do not blindly search for targets evenly across the entire bird's-eye view, but rather know which areas deserve priority attention and can continuously refer to the road structure when repeatedly updating queries.

[0035] The following detailed explanation of each step, in conjunction with the accompanying drawings, will be provided.

[0036] In step S1, the vehicle acquires multi-camera images and optional LiDAR point cloud data at the current moment. The acquired sensor data is input into a bird's-eye view encoder to obtain bird's-eye view features. These features represent environmental characteristics observed from a top-down view, including information on obstacles, road structures, and the surrounding environment. Bird's-eye view encoding can employ an encoder structure based on convolutional neural networks or Transformers, projecting and aggregating multi-view image features or point cloud features onto a bird's-eye view grid.

[0037] In step S2, based on the vehicle positioning results, map elements within the perception range of the high-precision map are extracted. The perception range can be set as a rectangular area of ​​60 meters in front and behind, and 30 meters to the left and right in the vehicle's coordinate system. The extracted map elements, such as lane lines, road boundaries, stop lines, and zebra crossing boundaries, are converted to the vehicle's current coordinate system to form a high-precision map vector polyline set. Each polyline in the vector polyline set includes several valid points and a corresponding polyline category label, and each polyline retains its two-dimensional coordinate point sequence and category label.

[0038] In this embodiment, the map polyline category is not limited to lane lines, road boundaries, stop lines, and zebra crossings, but may also include map elements such as lane center lines, road connections, traffic light stop lines, and intersection area boundaries. Any vector line element that can reflect the direction of road structure or traffic rule constraints falls within the scope of the polylines described in this invention.

[0039] In some embodiments, step S2 involves obtaining a set of high-precision map vector polylines corresponding to the scene to be detected, including: Based on the vehicle positioning results, map elements within the perception range of the high-precision map are extracted; the perception range is defined as a rectangular area within a first preset distance in front and behind and a second preset distance in the left and right under the vehicle coordinate system. The map elements include lane lines, road boundaries, stop lines, zebra crossing boundaries, lane center lines, road connections, traffic light stop lines, and intersection area boundaries. The captured map elements are converted to the vehicle's current coordinate system to form a high-precision map vector polyline set.

[0040] Step S3 encodes each polyline in the vector polyline set to generate a map token of the same dimension as the detection query, thus obtaining a map token set.

[0041] To facilitate batch processing by the neural network, each polyline is padded with a fixed number of points, with invalid values ​​used to fill any gaps. Specifically, the number of polylines in the vector polyline set is padded to a fixed number (e.g., a maximum of 128 polylines per frame), and the number of valid points in each polyline is padded to a fixed number (e.g., a maximum of 32 points per polyline), with invalid values ​​used to fill any gaps, generating a validity mask. This validity mask is used to mark whether each polyline is a real and valid map feature.

[0042] As a preferred implementation method, such as Figure 2 As shown, step S3 includes: Step S3.1: Encode each valid point in the polyline to obtain a geometric description at the polyline level.

[0043] Specifically, for each valid point in the polyline, one or more of its two-dimensional coordinates, differences between adjacent points, and point sequence position codes are used as point-level input features, which are then processed by a point-by-point multilayer perceptron to extract local geometric features. The point-by-point multilayer perceptron may include two or three linear layers, with a normalization layer and activation function added in between. These local geometric features characterize the direction, curvature, and continuity of the polyline within a local area, thereby distinguishing different geometric shapes such as straight segments, curved segments, and sharp bends. They are key intermediate features for subsequent pooling and aggregation into a polyline-level geometric description.

[0044] Based on the validity mask, max pooling, mean pooling, or attention pooling is performed only on valid points to obtain a polyline-level geometric description. The pooling operation aggregates a variable number of local geometric features into a fixed-length polyline-level feature vector, enabling the network to grasp the overall geometric shape of the entire polyline.

[0045] Step S3.2: Perform category embedding on the polyline category labels to obtain the map semantic type, enabling the network to distinguish different map objects such as lane lines, road boundaries, and zebra crossings. The map semantic type provides the network with semantic priors for the polylines; for example, lane lines indicate the boundaries of areas where targets may appear, and zebra crossings indicate the possible locations of pedestrians.

[0046] Step S3.3: Encode the starting valid points of the polyline to obtain spatial location information, enabling the network to know approximately which area the map object is located in the bird's-eye view. In one implementation, the location encoding can use sine / cosine encoding or learnable location embedding; in another implementation, normalized coordinate values ​​can be used directly as the location encoding.

[0047] Step S3.4: Concatenate and map the geometric description, map semantic type, and spatial location information to the target dimension to obtain a map token consistent with the detection query dimension.

[0048] In one example configuration, the detection query dimension is 256, and the map token dimension is also set to 256. If a maximum of 128 map polylines are retained per frame, then each sample can obtain a map token set with a shape of 128×256. For invalid polylines obtained through padding, their validity mask is invalid, and they are not included in the query reading in subsequent cross-attention calculations.

[0049] Step S3.5: Traverse each polyline in the vector polyline set, repeating steps S3.1 to S3.4 to obtain the map token set.

[0050] The above map token encoding method is not limited to multilayer perceptron pooling; it can also use one-dimensional convolution, Transformer encoder, graph neural network, or encoder based on piecewise linear topology.

[0051] Through the above processing, each map polyline is not simply synthesized into a single, blurry map feature map, but is instead saved as a map token as an independent map object. During subsequent decoding, the detection query can selectively read certain map objects based on relevance, such as lane lines or road boundaries near its own location, thus enabling the transfer of map information from the input feature layer to the query inference layer.

[0052] In step S4, road-related map features are extracted from the high-precision map to generate a road surface mask on the bird's-eye view. The road-related map features include lane lines, zebra crossings, road boundaries, and / or map features that can represent drivable areas.

[0053] Specifically, the extracted lane lines, road boundaries, zebra crossings, stop lines, or lane center lines are projected onto a predefined bird's-eye view grid coordinate system. A road area hint map, or road surface mask, is then generated based on the area within a certain radius near the map lines. The road surface mask does not need to be completely identical to the actual road surface; its function is to mark the proximity of each grid position in the bird's-eye view grid coordinate system to the road structure, indicating which areas are more likely to contain the target to be detected. Therefore, each grid position in the road surface mask has a numerical value; a higher value indicates that the position is closer to the relevant road map structure.

[0054] Road surface masks can be obtained by expanding map lines, or generated from lane surfaces, polygonal road regions, historical trajectory heatmaps, or map semantic grids. For example, in one implementation, map lines are expanded into a circle with a radius of 0.5 meters to form a strip-shaped area; in another implementation, predefined lane surface polygonal regions in a high-precision map are directly used as road surface masks.

[0055] In some embodiments, in step S4, Road-related map elements are extracted from high-precision maps, projected onto a predefined bird's-eye view grid coordinate system, and a road area hint map, i.e. a road surface mask, is generated based on the area within a preset radius near the map lines.

[0056] In step S5, such as Figure 3 As shown, dynamic sampling based on road surface masks includes: First, a road surface mask value threshold τ is set. Positions in the road surface mask where the mask value is lower than the threshold τ are set to zero, filtering out positions far from the road structure and retaining only candidate regions highly correlated with it. Second, the remaining positions are normalized to obtain a sampleable probability distribution, ensuring a clear probabilistic basis for the sampling process. Then, a predetermined number of bird's-eye view grid points are sampled from the probability distribution according to a preset query count (e.g., 300) as initial road sampling query points. Finally, random perturbations are added to the coordinates of each initial road sampling query point to obtain the final road sampling query points, avoiding complete overlap of multiple query starting points and increasing sampling diversity.

[0057] The mixing method for the road sampling query points and uniform prior query points can be any one of linear mixing, splicing, grouping, gating selection, or gradually adjusting the mixing ratio according to the training stage.

[0058] Taking linear hybrid as an example, first, the number of road sampling query points and uniform prior query points is unified to the same, and they are paired one by one according to a preset sorting, indexing, or nearest neighbor matching rule; linear interpolation is then performed on the two-dimensional bird's-eye view coordinates of each pair of corresponding query points to satisfy: ; in: For uniform prior query points; These are road sampling query points obtained based on road surface mask sampling; represents the prior mixing coefficients.

[0059] In this context, each query point is represented as a two-dimensional bird's-eye view coordinate with the same dimension as the detection query reference point. Each term in the formula represents the paired query point coordinates or a sequence of coordinates of the same shape. α is the prior mixing coefficient, with a value range of 0 ≤ α ≤ 1. For any pair of uniform prior query points and road sampling query points, the linear mixing result is the coordinate interpolation point between them. When α = 1, the final detection query point equals the uniform prior query point; when α = 0, the final detection query point equals the road sampling query point; when 0 < α < 1, the final detection query point is located between the corresponding uniform prior query point and the road sampling query point. The preset pairing rules can be pairing with the same index, pairing after spatial location sorting, or nearest neighbor matching.

[0060] In one example configuration, It can be set between 0.3 and 0.7.

[0061] For scenarios where targets, such as urban roads, intersections, and curves, are mostly located near roads, smaller targets... This helps prioritize queries that focus on road areas; for scenarios where the map is incomplete or the target may appear in atypical areas, a larger [map / database]... This helps retain stronger global search capabilities.

[0062] The mixing coefficient setting ensures that the query starting point of this invention retains the original uniform distribution prior while incorporating road structure priors from high-precision maps, thus balancing road guidance and global coverage. The mixing method for dynamic and uniform reference points is not limited to linear mixing; it can also employ methods such as stitching, grouping, gating selection, or gradually adjusting the mixing ratio according to the training phase.

[0063] In some embodiments, in step S5, when the mixing method of road sampling query points and uniform prior query points is linear mixing, the road sampling query points and uniform prior query points are mixed to obtain the final detection query points, as shown below: , In the formula, This is the final detection and query point. To ensure uniform prior query points, These are road sampling query points obtained based on road surface mask sampling. represents the prior mixing coefficients.

[0064] In step S6, such as Figure 4 As shown, map token cross-attention is performed on the query at each layer of the detector decoder, satisfying: in, This is the query vector output by the i-th layer query decoder; Let be the update query vector output by the i-th layer decoder after being guided by map token cross attention; This is a cross-attention function used to calculate the correlation between the query vector and each token in the map token set, and to aggregate map information accordingly. The set of map tokens serves as the key and value for the cross-attention function, where the key is used to calculate the attention weight with the query, and the value is used to aggregate map information based on the attention weight. This is a validity mask used to indicate whether each polyline is a valid map feature. Inject coefficients into the map residuals to control the strength of map information overlay onto the query vector.

[0065] In each layer of the detection decoder, the following sequence can be followed: First, environmental information is obtained from the bird's-eye view features, and visual or point cloud information from the sensor is obtained. Secondly, query intermediate results that predict or correct the target based on the current state, such as the target's location, size, and orientation; Then, the query reads the map token set through cross-attention to obtain map object information related to the current query. Specifically, the query vector calculates a relevance score with each token in the map token set, and the map information carried by each token is weighted and aggregated according to the relevance score to obtain map context information. During the attention calculation process, invalid polylines are masked according to the validity mask, so that the query only reads valid map tokens and avoids interference from invalid fill data on query updates; Finally, the map cross-attention output is added to the query state as a residual, and the injection intensity can be controlled by normalization layers, feedforward networks, or gating coefficients.

[0066] By repeatedly performing map token interactions at each layer of the detection decoder, queries can continuously refine their judgments across multiple iterations. For example, if a query initially falls near a road, it can identify suspected vehicle targets based on visual features, and then confirm the relationship between the location and lane lines, road boundaries, or intersection structures using map tokens, thereby improving the stability of positioning in complex road scenarios.

[0067] The detection decoder can share map attention parameters among different layers, or set them independently. When different layers share parameters, the number of model parameters can be reduced, thus lowering the risk of overfitting; when each layer sets its own parameters, different map attention modes can be assigned to different layers, improving the model's expressive power.

[0068] Map token interaction can be inserted into every layer of the detection decoder, or into some key layers. When inserting into some key layers, map guidance can be introduced at the most critical decoding stages of map information (such as the first few layers or the last few layers), reducing computational overhead while ensuring detection accuracy.

[0069] Map injection strength can be set to a fixed coefficient or adaptively adjusted based on map confidence, location confidence, distance between the query and map objects, or scene type. For example, increasing the location confidence level can be beneficial when the location confidence is high. To enhance map guidance; when location confidence is low, reduce To prevent incorrect map alignment information from misleading detection.

[0070] After several layers of detection decoder iterations, the updated query is used for 3D bounding box regression and category prediction, outputting the location, size, orientation, and category of targets such as vehicles, pedestrians, and cyclists.

[0071] It is worth noting that during the training phase, this invention can use the same detection loss as existing query-based 3D object detection networks, such as classification loss, 3D bounding box regression loss, orientation angle loss, and matching loss. Map token encoding, dynamic sampling to generate the final detection query point, and map token cross-attention operation can be trained end-to-end along with the 3D object detection network, or they can be initialized using existing detection network weights first, and then fine-tuned by adding a map guidance module.

[0072] During the inference phase, the vehicle reads a local high-precision map based on its current location, obtains a set of high-precision map vector polylines, and generates a road surface mask and a set of map tokens. The 3D target detection network outputs 3D target detection results based on sensor data and map guidance. Since map token interaction only adds one attention calculation for the map token set in each layer of the detection decoder, and the number of map polylines is controlled by a fixed upper limit, the computational overhead can be controlled by limiting the number of map objects, sharing map attention parameters, or reducing the number of attention heads.

[0073] When high-precision maps are missing, positioning is abnormal, map polylines are empty, or map quality does not meet requirements, the following rollback methods can be used: All validity masks are set to invalid, making the map token cross-attention output close to zero, and map information cannot be queried or read; the dynamic sampling method of road surface mask is replaced with uniform random sampling or learnable prior sampling; the map residual injection coefficient β is set to zero, or the map branch weight is reduced through a gating network; the bird's-eye view encoding and detection decoding process continue to work, so that the system degenerates into a 3D target detection process without map guidance.

[0074] Therefore, this invention can improve detection and query efficiency and road structure perception capabilities when high-precision maps are available, and can also maintain basic detection capabilities when maps are unavailable, making it suitable for different operating conditions in actual autonomous driving systems.

[0075] In other embodiments, the present invention discloses a map token-guided 3D target detection device, comprising: The bird's-eye view encoding module is used to encode the collected sensor data into bird's-eye view features. The map acquisition module is used to acquire a set of high-precision map vector polylines corresponding to the scene to be detected. Each polyline in the vector polyline set includes: several valid points and a corresponding polyline category label; The map token encoding module is used to encode each polyline in the vector polyline set, generate map tokens that can be directly read by queries, and obtain a map token set; The road surface mask generation module is used to extract road-related map features from high-precision maps and generate road surface masks on bird's-eye views. The query point generation module is used to perform dynamic sampling based on the road surface mask to generate road sampling query points, and then mix the road sampling query points with uniform prior query points to obtain the final detection query points. The detection decoding module is used to feed the bird's-eye view features into the detection decoder, using the final detection query point as the starting reference point for the detection query. At each layer or some key layers of the detection decoder, map token cross-attention operation is performed on the query, so that the query reads map information in the map token set during the iteration process. After the detection decoder iterates, the 3D target detection result is output.

[0076] Furthermore, it should be noted that the map token-guided 3D target detection device provided in the above embodiments is only illustrated by the division of the above functional modules when determining the dominant vertex. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the map token-guided 3D target detection device can be divided into different functional modules to complete all or part of the functions described above.

[0077] Furthermore, the map token-guided 3D target detection device and the map token-guided 3D target detection method provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0078] In other embodiments, the present invention discloses a computing device comprising: One or more processors; Memory; And one or more programs, wherein the one or more programs are stored in memory and configured to be executed by one or more processors, and the one or more programs include instructions for any of the map token-guided 3D object detection methods described above.

[0079] In other embodiments, the present invention discloses a storage medium storing one or more computer-readable programs, the programs including instructions adapted to be loaded by memory and executed in any of the map token-guided 3D object detection methods described above.

[0080] This invention discloses a three-dimensional target detection method and apparatus based on map token guidance, which has the following beneficial effects: First, this invention encodes high-precision map vector polylines into map tokens that can be directly read by queries, so that map information no longer stays only at the network input feature layer, but enters the query inference layer and participates in the target detection process, fundamentally solving the technical problem that map information is difficult to directly participate in query generation.

[0081] Second, this invention performs map token cross-attention operations on the query at each or some key layers of the detection decoder, so that the query continuously obtains road geometry and semantic constraints during the iterative update process, rather than using the map only once at the input stage, effectively solving the problem that it is difficult to obtain a clear road network structure for long-distance queries.

[0082] Third, the present invention generates road sampling query points based on road surface masking through dynamic sampling, which enables the query to be preferentially concentrated on road-related areas from the initialization stage, thereby improving the utilization rate of effective queries and improving the defect of query initialization being disconnected from road geometry.

[0083] Fourth, this invention mixes road sampling query points with uniform prior query points, which not only uses map priors to guide queries to road areas, but also retains global search capabilities, avoiding the problem of insufficient global coverage caused by excessive concentration of queries in road areas.

[0084] Fifth, this method does not require rewriting or modifying the overall structure of the basic detection decoder; it can be implemented simply by adding map token encoding, sampling, and map token cross-attention related algorithms, thus possessing good engineering compatibility.

[0085] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the present invention. Various changes and modifications can be made to the present invention without departing from its spirit and scope. All such changes and modifications fall within the scope of the present invention as claimed, which is defined by the appended claims and their equivalents.

Claims

1. A 3D target detection method based on map token guidance, characterized in that, include: Step S1: Encode the collected sensor data into bird's-eye view features; Step S2: Obtain the set of high-precision map vector polylines corresponding to the scene to be detected. Each polyline in the set of vector polylines includes: several valid points and corresponding polyline category labels; Step S3: Encode each polyline in the vector polyline set to generate a map token that can be directly read by the query, and obtain a map token set; Step S4: Extract road-related map elements from the high-precision map and generate road surface masks on the bird's-eye view; Step S5: Perform dynamic sampling based on the road surface mask to generate road sampling query points, and then mix the road sampling query points with uniform prior query points to obtain the final detection query points; Step S6: The bird's-eye view features are fed into the detection decoder. The final detection query point is used as the starting reference point for the detection query. Map token cross-attention operation is performed on the query at each layer or some key layers of the detection decoder, so that the query reads the map information in the map token set during the iteration process. After the detection decoder iterates, the three-dimensional target detection result is output.

2. The method according to claim 1, characterized in that, Step S3 includes: Step S3.1: Encode each valid point in the polyline to obtain a geometric description at the polyline level; Step S3.2: Perform category embedding on the polyline category labels to obtain the map semantic type; Step S3.3: Encode the position of the starting valid point of the polyline to obtain spatial position information; Step S3.4: Concatenate and map the geometric description, the map semantic type, and the spatial location information to the target dimension to obtain a map token consistent with the detection query dimension; Step S3.5: Traverse each polyline in the vector polyline set, repeating steps S3.1 to S3.4 to obtain the map token set.

3. The method according to claim 1, characterized in that, Before performing map token encoding on each polyline in the vector polyline set in step S3, the method further includes: The number of polylines in the vector polyline set is filled to a fixed number, and the number of valid points in each polyline is filled to a fixed number. Any insufficient points are padded with invalid values ​​to generate a validity mask. The validity mask is used to mark whether each polyline is a real and valid map feature.

4. The method according to claim 1, characterized in that, In step S5, dynamic sampling based on the road surface mask includes: The positions in the road surface mask below the threshold are set to zero, and the remaining positions are normalized to obtain the sampleable probability distribution; According to the preset number of queries, a corresponding number of bird's-eye view grid points are sampled from the probability distribution as initial road sampling query points; Random perturbations are added to the coordinates of each of the initial road sampling query points to obtain the final road sampling query points.

5. The method according to claim 1, characterized in that, The mixing method for the road sampling query points and uniform prior query points can be any one of linear mixing, splicing, grouping, gating selection, or gradually adjusting the mixing ratio according to the training stage.

6. The method according to claim 1, characterized in that, In step S6, a map token cross-attention operation is performed on the query at each layer or part of the key layers of the detection decoder, satisfying: in, This is the query vector output by the i-th layer query decoder; Let be the update query vector output by the i-th layer decoder after being guided by map token cross attention; This is a cross-attention function used to calculate the correlation between the query vector and each token in the map token set, and to aggregate map information accordingly. This is a validity mask used to indicate whether each polyline is a valid map feature. A set of map tokens, serving as the key and value for the cross-attention function; Inject coefficients into the map residuals.

7. The method according to claim 1, characterized in that, The map attention parameters can be shared between different layers of the detection decoder, or set independently.

8. A 3D target detection device based on map token guidance, characterized in that, include: The bird's-eye view encoding module is used to encode the collected sensor data into bird's-eye view features. The map acquisition module is used to acquire a set of high-precision map vector polylines corresponding to the scene to be detected. Each polyline in the set of vector polylines includes: several valid points and a corresponding polyline category label; The map token encoding module is used to encode each polyline in the vector polyline set to generate a map token that can be directly read by the query, thus obtaining a map token set; The road surface mask generation module is used to extract road-related map elements from the high-precision map and generate road surface masks on the bird's-eye view. The query point generation module is used to perform dynamic sampling based on the road surface mask to generate road sampling query points, and then mix the road sampling query points with uniform prior query points to obtain the final detection query points. The detection decoding module is used to send the bird's-eye view features into the detection decoder, use the final detection query point as the starting reference point for the detection query, and perform map token cross-attention operation on the query at each layer or some key layers of the detection decoder, so that the query reads the map information in the map token set during the iteration process; after the detection decoder iterates, it outputs the three-dimensional target detection result.

9. A computing device, characterized in that, include: One or more processors; Memory; And one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by one or more processors, and the one or more programs include instructions for the map token-guided 3D object detection method according to any one of claims 1-7.

10. A storage medium, characterized in that, The storage medium stores one or more computer-readable programs, the programs including instructions adapted to be loaded by memory and executed as described in any of claims 1-7 for the map token-guided 3D target detection method.

Images