High-precision map alignment fusion method and system suitable for multi-view target detection

Abstract

The application provides a high-precision map alignment fusion method and system suitable for multi-view target detection, which comprises the following steps: acquiring image features of multi-view images to be detected at a given time; acquiring map features of a high-precision map within a vehicle perception range of a vehicle position and direction recorded at a given time; splicing the image features and the map features to obtain cross-modal features; performing amplitude adjustment and map information strengthening processing on the cross-modal features to obtain fusion features; and performing target detection on the fusion features to obtain a target detection result. The high-precision map alignment fusion method suitable for multi-view target detection provided by the application improves the accuracy of multi-view three-dimensional target detection by performing cross-modal fusion on multi-view images and a high-precision map, and is suitable for high-precision vehicle target detection and recognition in the automatic driving technology of multi-view cameras.

Description

Technical Field

[0001] This application relates to the field of video image processing technology, specifically to a high-precision map alignment and fusion method and system applicable to multi-view target detection. Background Technology

[0002] High-level autonomous driving has garnered widespread attention due to its ability to reduce driver workload and improve driving safety. As an indispensable part of perception in autonomous vehicles, 3D object detection aims to predict the geometric and semantic information of key targets around the vehicle. Reliable detection results are crucial for downstream tasks such as target tracking, trajectory prediction, and motion planning. Multimodal fusion, which captures signals from multiple sensors and enhances detection quality through complementarity, is of great significance for advancing high-level autonomous driving. However, high-precision maps developed for autonomous driving have not received sufficient attention, and research on object detection through the fusion of high-precision maps with other sensors, particularly multi-view images, is lacking.

[0003] The research challenge in fusing high-precision maps and multi-view images (camera-map fusion) lies in the significant differences between the two. The two modalities of data exist at different perspectives, semantic levels, and amplitude scales, hindering the construction of consistent feature representations. The fusion of multi-view images and LiDAR (camera-radar fusion) is the most mature, with various bimodal fusion frameworks proposed, employing different stages, feature representations, and operators. Preliminary research has been conducted on the fusion of LiDAR and high-precision maps (radar-map fusion), exploring strategies for the representation, encoding, and fusion of high-precision maps.

[0004] Nevertheless, the differences between different modal data make it impossible to migrate existing multimodal frameworks to camera map fusion. On the other hand, pure vision 3D object detection has gained widespread attention and developed rapidly due to its low deployment cost, but it cannot overcome the inherent limitations of single-modal methods; and there is currently a lack of camera map fusion methods. Summary of the Invention

[0005] This application provides a high-precision map alignment and fusion method and system suitable for multi-view target detection, which can solve the technical problem of the lack of camera map fusion methods for 3D target detection in the prior art.

[0006] Firstly, this application provides a high-precision map alignment and fusion method suitable for multi-view target detection, comprising the following steps:

[0007] Obtain image features of a multi-view image to be detected at a given time;

[0008] Obtain map features of a high-precision map within the vehicle's perception range, based on the vehicle's position and orientation recorded at a given time.

[0009] By stitching together image features and map features, cross-modal features can be obtained;

[0010] Amplitude adjustment and map information enhancement processing are performed on cross-modal features to obtain fused features;

[0011] Target detection is performed on the fused features to obtain the target detection results.

[0012] In conjunction with the first aspect, in one implementation, the map features of the high-precision map within the vehicle's perception range, recording the vehicle's position and direction at a given time, include:

[0013] Based on the vehicle's position and direction recorded at a given time, a local high-precision map within the vehicle's perception range is cropped from the high-precision map;

[0014] Obtain a local map representation of a high-definition local map;

[0015] Obtain image labeling information from the local map representation;

[0016] Add image labeling information to the local map representation to obtain a balanced map representation;

[0017] Hollow convolution processes balanced maps and represents them as map features.

[0018] In conjunction with the first aspect, in one implementation, the local map representation of acquiring the local high-precision map includes:

[0019] A rasterized local high-precision map consists of multiple rasters;

[0020] For each semantic layer of the local high-precision map, the range of the semantic annotation of each grid center within the semantic annotation range of the semantic layer is compared layer by layer. The binary image containing the semantic annotation range determination information of each semantic layer is used as the local map representation.

[0021] In conjunction with the first aspect, in one embodiment, obtaining the image tagging information of the local map representation includes:

[0022] Construct a new binary image;

[0023] Obtain the location information of all semantic layer images in the local map representation;

[0024] If an image in the local map representation is at the same location as a new binary image, then the image labeling information that marks the image in the new binary image is the first label;

[0025] If an image in the partial attached drawing is located at a different position than the new binary image, then the image labeling information that marks the image in the new binary image is the second label.

[0026] In conjunction with the first aspect, in one implementation, the step of adding image labeling information to the local map representation to obtain a balanced map representation includes:

[0027] By stitching together the channels of the binary images of each semantic layer in the local map representation and a new binary image containing image labeling information, a balanced map representation is obtained.

[0028] In conjunction with the first aspect, in one implementation, the dilated convolution processing of the balanced map representation as map features includes:

[0029] For the balanced map representation, the first map extraction features are obtained through dilated convolutional layers, SE modules, and downsampling convolutional layers. The feature dimension of the dilated convolutional layer is the first dilated convolution feature dimension, and the feature dimension of the downsampling convolutional layer is the first downsampling convolution feature dimension.

[0030] For the first map feature extraction, the second map feature extraction is obtained through dilated convolutional layer, SE module, and downsampling convolutional layer. The feature dimension of the dilated convolutional layer is the second dilated convolution feature dimension, and the feature dimension of the downsampling convolutional layer is the second downsampling convolution feature dimension.

[0031] For the second map feature extraction, the third map feature extraction is obtained through dilated convolutional layer, SE module, and local convolutional layer. The feature dimension of the dilated convolutional layer is the same as the feature dimension of the second dilated convolution, and the feature dimension of the local convolutional layer is the same as the feature dimension of the second dilated convolution, thus obtaining the third map feature extraction.

[0032] The features extracted from the second map and the features extracted from the third map are added element by element to obtain the final map features;

[0033] Wherein, the dimension of the first dilated convolution feature is less than the dimension of the second dilated convolution feature, which is less than the dimension of the third dilated convolution feature, and the dimension of the first downsampled convolution feature is less than the dimension of the second downsampled convolution feature. In conjunction with the first aspect, in one implementation, the step of performing amplitude adjustment and map information enhancement processing on the cross-modal features to obtain fused features includes:

[0034] Amplitude adjustment is performed on cross-modal features to obtain channel attention;

[0035] Map information enhancement processing is applied to cross-modal features to obtain spatial attention;

[0036] The channel attention and spatial attention are multiplied and then normalized using sigmoid to obtain the fused features.

[0037] In conjunction with the first aspect, in one implementation, the amplitude adjustment of cross-modal features to obtain channel attention includes:

[0038] Global average pooling is applied to cross-modal features, and the features are flattened to obtain feature vectors.

[0039] The feature vector is compressed by a fully connected layer and then normalized and ReLU activated to obtain the compressed feature vector.

[0040] Channel attention is obtained by restoring the number of channels in the compressed feature vector through a fully connected layer.

[0041] In conjunction with the first aspect, in one implementation, the step of performing map information enhancement processing on cross-modal features to obtain spatial attention includes:

[0042] For cross-modal features, convolution compresses the feature channels and then performs normalization and ReLU activation to obtain the first feature;

[0043] For the first feature, dilated convolution is performed to extract the feature, and then normalization and ReLU activation are performed to obtain the second feature;

[0044] For the second feature, dilated convolution is performed to extract the feature, and then normalization and ReLU activation are performed to obtain the third feature;

[0045] Compress the third special channel to eliminate channel dimension and obtain spatial attention.

[0046] Secondly, this application provides a high-precision map alignment and fusion system suitable for multi-view target detection, comprising:

[0047] The image feature acquisition module is used to acquire the image features of the multi-view image to be detected at a given time.

[0048] The map feature acquisition module is used to acquire map features of a high-precision map within the vehicle's perception range, based on the vehicle's position and direction recorded at a given time.

[0049] A cross-modal feature acquisition module is communicatively connected to the image feature acquisition module and the map feature acquisition module, and is used to stitch together image features and map features to acquire cross-modal features;

[0050] The fusion feature acquisition module is communicatively connected to the cross-modal feature acquisition module and is used to perform amplitude adjustment and map information enhancement processing on the cross-modal features to acquire fusion features;

[0051] The target detection module is communicatively connected to the fusion feature acquisition module and is used to perform target detection on the fusion features and obtain target detection results.

[0052] The beneficial effects of the technical solutions provided in this application include at least the following:

[0053] The high-precision map alignment and fusion method for multi-view target detection provided in this application improves the accuracy of multi-view 3D target detection by cross-modal fusion of multi-view images and high-precision maps, and is applicable to vehicle target detection and recognition in autonomous driving technology using multi-view cameras. Attached Figure Description

[0054] Figure 1 This is a schematic diagram of the method flow for a high-precision map alignment and fusion method applicable to multi-view target detection provided in an embodiment of this application;

[0055] Figure 2 This is a flowchart of a high-precision map embedding method for multi-view 3D target detection, provided in the high-precision map alignment and fusion method for multi-view target detection in the embodiments of this application.

[0056] Figure 3 A flowchart of the high-precision map semantic alignment method in the high-precision map alignment and fusion method for multi-view target detection provided in the embodiments of this application;

[0057] Figure 4 A schematic diagram of the high-precision map amplitude alignment and fusion method in the high-precision map alignment and fusion method for multi-view target detection provided in the embodiments of this application;

[0058] Figure 5 This is a functional block diagram of a high-precision map alignment and fusion system for multi-view target detection provided in an embodiment of this application. Detailed Implementation

[0059] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0060] The terms "comprising" and "having," and any variations thereof, in the specification, claims, and accompanying drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus. The terms "first," "second," and "third," etc., are used to distinguish different objects, etc., and do not indicate a sequence, nor do they limit "first," "second," and "third" to different types.

[0061] In the description of the embodiments of this application, terms such as "exemplary," "for example," or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary," "for example," or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary," "for example," or "for instance" is intended to present the relevant concepts in a concrete manner.

[0062] In the description of the embodiments of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. The "and / or" in the text is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of this application, "multiple" means two or more.

[0063] In some processes described in the embodiments of this application, multiple operations or steps are included in a specific order. However, it should be understood that these operations or steps may not be executed in the order they appear in the embodiments of this application, or they may be executed in parallel. The sequence number of the operation is only used to distinguish different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed sequentially or in parallel, and these operations or steps may be combined.

[0064] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0065] Firstly, please refer to Figure 1 and Figure 2 This application provides a high-precision map alignment and fusion method suitable for multi-view target detection, including the following steps:

[0066] Step S100: Obtain the image features of the multi-view image to be detected at a given time.

[0067] Step S200: Obtain the map features of a high-precision map within the vehicle's perception range, based on the vehicle's position and direction recorded at a given time.

[0068] Step S300: Combine image features and map features to obtain cross-modal features;

[0069] Step S400: Perform amplitude adjustment and map information enhancement processing on the cross-modal features to obtain fused features;

[0070] Step S500: Perform target detection on the fused features and obtain the target detection results.

[0071] The high-precision map alignment and fusion method for multi-view target detection provided in this application improves the accuracy of multi-view 3D target detection by cross-modal fusion of multi-view images and high-precision maps, and is applicable to vehicle target detection and recognition in autonomous driving technology using multi-view cameras.

[0072] In one embodiment, the high-precision map alignment and fusion method described above, which is suitable for multi-view target detection, is applied to vehicle target detection and recognition in autonomous driving scenarios. The experimental scenario comes from the large-scale autonomous driving dataset nuScenes.

[0073] In one embodiment, for a multi-view image to be detected at a given time, the image features of the multi-view aggregated bird's-eye view are extracted using a multi-view 3D target detection baseline. More specifically, step S100, obtaining the image features of the multi-view image to be detected at a given time, includes:

[0074] Step S110: Export and acquire multi-view images from the vehicle-mounted camera. Multi-view images generally refer to images from 6 perspectives.

[0075] Step S120: Using BEVDet as the baseline for multi-view 3D target detection, input the multi-view images into the Swin-Transformer backbone network for feature extraction to obtain single-view image features.

[0076] Step S130: The acquired single-view image features are transformed using LSS(Lift,Splat,Shoot) to obtain the multi-view aggregated image features.

[0077] In one embodiment, step S120, using BEVDet as the baseline for multi-view 3D object detection, inputs the multi-view images into the Swin-Transformer backbone network for feature extraction, and obtains single-view image features, is specifically implemented as follows:

[0078] Simultaneously, 3×512×1408 images from 6 viewpoints are input into the Swin-Transformer backbone network for feature extraction. The backbone network adopts the base setting, and the last two feature maps are output for each viewpoint. The width and height of the first layer are twice that of the second layer, while the number of channels is half that of the second layer.

[0079] The feature map of the next layer for each viewpoint is upsampled by bilinear interpolation to double its width and height, and then spliced ​​with the feature map of the previous layer along the channel to form a feature pyramid structure. The single-view image features with a size of 512×16×44 are extracted through two convolutional layers.

[0080] In one embodiment, step S130, which involves transforming the acquired single-view image features using LSS (What is the full name of LSS) to obtain multi-view aggregated image features, includes:

[0081] Step S131: Set the vehicle perception space range to 51.2m along the vehicle body direction and perpendicular to the vehicle body direction, with a side length of 102.4m. The spatial resolution of the voxel in the bird's-eye view is 0.4m. Based on this, preset the depth range corresponding to each feature pixel to be 1-60m, with a value interval of 1m.

[0082] Step S132: Based on the above settings, the LSS method is used to aggregate multi-view image features, specifically including:

[0083] A. For each viewpoint, use a convolutional network to predict the depth distribution of each feature pixel.

[0084] B. By utilizing the camera's intrinsic parameters and the camera's extrinsic parameters relative to the LiDAR, the feature pixels are mapped onto the LiDAR's three-dimensional space according to their depth distribution, presenting a feature view frustum.

[0085] C. Fill the feature frustums of all 6 perspectives within the same space, remove the parts outside the perceptual range, aggregate voxel features according to spatial resolution in the bird's-eye view, and compress and eliminate the height axis to obtain the multi-view image features in the bird's-eye view.

[0086] In one embodiment, step S200, obtaining map features of a high-precision map within the vehicle's perception range at a given time, including the vehicle's position and direction, includes:

[0087] Step S210: Based on the vehicle position and direction recorded at a given time, crop out a local high-precision map within the vehicle's perception range from the high-precision map;

[0088] Step S220: Obtain the local map representation of the local high-precision map;

[0089] Step S230: Obtain image marker information representing the local map;

[0090] Step S240: Append image labeling information to the local map representation to obtain a balanced map representation;

[0091] Step S250: Hollow convolution processing balances the map representation into map features.

[0092] In one embodiment, step S210, based on the vehicle position and direction recorded at a given time, involves cropping a local high-precision map within the vehicle's perception range from the high-precision map. Specifically, this is implemented as follows:

[0093] Given the vehicle's position and azimuth angle α in the world coordinate system at the current moment, construct a square with the vehicle's position as the geometric center and a sensing range of 102.4m as the side length. Rotate the square so that the orientation of its two sides is azimuth angle α. Crop a local high-precision map from the high-precision map of the area along the boundary of the rotated square.

[0094] In one embodiment, step S220, obtaining the local map representation of the local high-precision map, is specifically implemented as follows:

[0095] Step S221: Rasterize the local high-precision map into multiple grids. More specifically, rotate the local high-precision map counterclockwise by π / 2-α and divide it into 256×256 grids. The grid spatial resolution is the spatial resolution of the image feature voxels in the bird's-eye view, which is 0.4m.

[0096] Step S222: For each semantic layer of the local high-precision map, compare the range of the semantic annotation of each grid center within the semantic annotation range of the semantic layer layer. The image containing the semantic annotation range determination information of each semantic layer is a binary image, which is used as the local map representation.

[0097] In one embodiment, step S222 involves comparing the semantic annotation range of each raster center within the semantic annotation range of the local high-precision map layer by layer, identifying any memory limitations. The image containing the semantic annotation range determination information for each semantic layer is a binary image, which serves as a local map representation.

[0098] Select six semantic layers from the high-definition map: {drivable_area, ped_crossing, walkway, stop_line, carpark_area, divider}, and repeat the following comparison operation until a binary image of each semantic layer is constructed. i Let i = 1, 2, ..., 6, and let its set M be used as the local map representation, i.e., M = {l i ,i=1,2,…,6}:

[0099] For a selected semantic layer s of a local high-definition map, construct an image;

[0100] If the center (u,v) of a raster in the semantic layer s is within the semantic annotation range A, then the corresponding pixel in the newly constructed image of the semantic layer is set to 1; otherwise, it is set to 0, as follows:

[0101]

[0102] In the formula, (u,v)∈[0,4h)×[0,4w), It is the constructed corresponding binary image;

[0103] The image containing the semantic annotation range determination information of each semantic layer is used as a binary image, i.e., a local map representation.

[0104] It should be noted that in step S222, an image is constructed for each semantic layer, for a total of 6 images. Each semantic layer has a corresponding semantic annotation range, which is provided by the dataset in the high-precision map data.

[0105] In one embodiment, step S230, obtaining image tagging information of the local map representation, includes:

[0106] Step S231: Construct a new binary image;

[0107] Step S232: Obtain the location information of all semantic layer images in the local map representation;

[0108] Step S233A: If any image in the local map representation is at the same position as the new binary image, then the image labeling information of that image in the new binary image is the first label;

[0109] Step S233B: If any image in the local map representation and the new binary image are in different locations, then the image labeling information of that image in the new binary image is the second label.

[0110] In one embodiment, in a typical high-definition map, semantically labeled areas only account for a small portion, while unlabeled areas constitute the majority of the perceived area. In the high-definition maps of the four locations provided by nuScenes in this example, unlabeled areas account for 82%, 84%, 88%, and 90% of the perceived area, respectively. However, the lack of labeling causes the network to pay less attention to these areas, weakening the features of the targets within them. Therefore, it is necessary to incorporate unlabeled areas as cross-modal fusion features to enhance the target features. Step S231, constructing a new binary image, is specifically implemented as follows:

[0111] Construct a binary image l b For spatial locations (u,v)∈[0,256)×[0,256), if any image in set M has a value of 1 at the same location, then l b The value at this position is 0 otherwise, represented as:

[0112] l b (u,v)=1-(l1(u,v)⊕l2(u,v)⊕…⊕l n (u,v)).

[0113] In the formula, ⊕ represents Boolean sum.

[0114] In one embodiment, step S240, appending image marker information to the local map representation to obtain a balanced map representation, includes:

[0115] By concatenating the binary images of each semantic layer in the local map representation and a new binary image containing image labeling information along the channels, a balanced map representation is obtained, such as... Figure 3 As shown, the specific implementation is as follows:

[0116] The binary image l b Add to the local map representation M = {l1, l2, ..., l6}, and associate l1, l2, ..., l6 with l b By piecing together the data along the channels, a balanced map representation can be obtained. This step provides a direct representation for unlabeled areas, giving them equivalent physical meaning to labeled areas, thus prompting the network to focus on these regions. It also offers an additional benefit: the high proportion of unlabeled areas results in low information density in the initial high-precision map. (Binary image l) b The main area in the map image representation is added with a value of 1, and all coordinates are guaranteed to have at least one non-zero value. This significantly improves the information density of the balanced map representation, making it closer to the characteristics of multi-view images, which is beneficial for subsequent fusion.

[0117] In one embodiment, to establish long-range dependencies in the spatial domain, and considering that high-definition maps do not need to capture low-level visual patterns, dilated convolution encoding achieves better results. Encoding elevates the low-feature-dimensional high-definition map to a high-dimensional space, enriching its representation and semantically aligning it with high-density image features. Furthermore, [the following is unclear and likely incomplete: "for..."] Features were extracted again using dilated convolutional layers, SE modules, and downsampling convolutional layers. right Features are extracted using dilated convolutional layers, SE modules, and local convolutional layers. and Map features are obtained by adding elements one by one. More specifically, such as Figure 3 As shown, step S250, the dilated convolution process for balancing the map, represents the map features as follows:

[0118] Step S251: Represent the balanced map M b The first map features are extracted through dilated convolutional layers, SE modules, and downsampling convolutional layers. The dilated convolution is set to a 3×3 kernel, stride 1, and dilation factor 3, increasing the feature dimension to 16; the SE module adjusts the weights of each channel; the downsampling convolution is set to a 1×1 kernel and stride 2, halving the feature map size and increasing the feature dimension to 32. Normalization and ReLU activation are immediately applied after each convolutional layer. The resulting map features...

[0119] Step S252: Extract features from the first map. Repeat step S251 to extract features from the second map using a dilated convolutional layer, an SE module, and a downsampling convolutional layer. Unlike step S251, the feature dimension of the dilated convolutional layer is 32, and the feature dimension of the downsampling convolutional layer is 64.

[0120] Step S253: Extract features from the second map. Third-map features are extracted using dilated convolutional layers, SE modules, and local convolutional layers. In this configuration, dilated convolutions are set to 3×3 kernels with a stride of 1 and a dilation factor of 3, while maintaining a feature dimension of 64. The SE module adjusts the weights of each channel. Local convolutions are set to 1×1 kernels with a stride of 1, while all feature dimensions remain unchanged. Each convolutional layer is followed immediately by normalization and ReLU activation to obtain map features. and The shapes are exactly the same;

[0121] Step S254: Extract features from the second map. Feature extraction from third maps Add each element one by one to obtain the final map feature F. m .

[0122] The high-precision map alignment and fusion method for multi-view target detection provided in this application makes map features semantically closer to image features through high-precision map encoding.

[0123] In one embodiment, step S300, which involves stitching together image features and map features to obtain cross-modal features, is specifically implemented as follows:

[0124] The image features F obtained in step S100 c And the map features F obtained in step S200 m They have exactly the same shape and correspond to the same physical spatial range (102.5m) and spatial resolution (0.4m), i.e., image feature F. c and map features F m It has a strict spatial correspondence in both width and height dimensions. Therefore, the image features F can be... c and map features F mBy splicing along the channels, cross-modal features are obtained.

[0125] In one embodiment, step S400, which involves amplitude adjustment and map information enhancement processing of cross-modal features to obtain fused features, includes:

[0126] Step S410: For cross-modal features Amplitude adjustment is performed on cross-modal features to obtain channel attention chn(F), which is expressed as:

[0127] chn(F) = MLP(AvgPool(F));

[0128] Step S420: Compress the feature dimension of F using convolution, perform map information enhancement processing on the cross-modal features, and obtain the spatial attention spt(F), represented as:

[0129] spt(F) = conv(dil) 2 (reduce(F)));

[0130] Step S430: Multiply the channel attention chn(F) and spatial attention spt(F) and perform sigmoid normalization to obtain the fused feature F. f .

[0131] This application aims to provide a lightweight and efficient high-precision map alignment and fusion method for autonomous driving, addressing differences in viewpoint, semantics, and amplitude. This method improves the accuracy of multi-view 3D object detection and is applicable to vision-based autonomous driving solutions. By utilizing channel attention and spatial attention for cross-modal feature fusion, it alleviates the differences in viewpoint, semantics, and amplitude between images and maps, enabling the integration of high-precision maps into multi-view 3D object detection.

[0132] In one embodiment, for cross-modal features It simply stitches together camera and map modal features along the channel dimension. A two-way problem exists during fusion: looking forward, the features of the two modalities may have significant amplitude differences, the specific value of which cannot be known in advance; looking backward, the features of the two modalities may have significantly different contributions to subsequent detection tasks, the specific value of which cannot be known in advance. This lack of prior knowledge necessitates the design of a learnable adaptive amplitude adjustment module. For the separation of different modal channels, this manifests as channel attention. Step S410, adjusting the amplitude of cross-modal features to obtain channel attention, includes:

[0133] Step S411: Perform global average pooling on the cross-modal feature F to obtain... Flattened into feature vectors

[0134] Step S412: Compress the feature vector through a fully connected layer. A vector of length 1 / 16 of the input is obtained, and then normalized and ReLU activated to obtain the compressed feature vector.

[0135] Step S413: Recover the compressed feature vector through a fully connected layer. The number of channels is used to obtain the channel attention chn(F)∈R. 128 .

[0136] In one embodiment, noting the unique properties of high-definition maps, the fusion of cross-modal features F requires further customized design. While encoding high-definition maps is essential, it inevitably leads to some loss of original information. Considering that the original information is ground truth annotation, rather than low-level visual patterns such as color and coordinates, it is necessary to mitigate the impact of this loss to fully utilize the high-definition map. Inspired by location embedding in point cloud feature extraction, a spatial attention-based approach is designed to compensate for this loss by emphasizing the geographic information of the features. Please refer to [reference needed]. Figure 4 Step S420, performing map information enhancement processing on cross-modal features to obtain spatial attention, includes:

[0137] Step S421: For the cross-modal feature F, use a 1×1 convolution kernel with a stride of 1 to compress the feature channels to 1 / 16 of the input, and then perform normalization and ReLU activation to obtain the first feature.

[0138] Step S422, for the first feature Features are extracted using dilated convolution, and then normalized and ReLU activated to obtain the second feature.

[0139] Step S423: Repeatedly perform dilated convolution to extract features from the second feature, and then perform normalization and ReLU activation to obtain the third feature.

[0140] Step S424: Compress the third feature channel to eliminate channel dimension and obtain spatial attention.

[0141] In one embodiment, step S430, multiplying the channel attention and spatial attention and performing sigmoid normalization to obtain fused features, includes:

[0142] The channel attention chn(F) obtained in step S410 and the spatial attention spt(F) obtained in step S420 are applied to F. chn(F) and spt(F) are multiplied and normalized using the sigmoid function. The resulting attention is then multiplied element-wise with F. The resulting feature is added to the initial F, and the fused feature F is output. f , is represented as:

[0143] F f =F+sigmoid(chn(F)×spt(F))×F.

[0144] In a more specific embodiment, step S430, multiplying the channel attention and spatial attention and performing sigmoid normalization to obtain the fused features, is specifically implemented as follows:

[0145] The dimensions of the extended channel attention chn(F) constitute W c The size of the extended spatial attention spt(F) constitutes W s . W c (u,v,:)=chn(F),W s (:,:,i)=spt(F), v∈[0,w),i∈[0,c);where h,w,c all take the value 64.

[0146] W c With W s Element-wise multiplication, and then using the sigmoid function to obtain the attention W = sigmoid(W c ×W s );

[0147] The cross-modal feature F is multiplied element-wise with the attention W, and then added to F itself to obtain the final fused feature F. f , represented as F f =F + W × F.

[0148] In one embodiment, the fused features obtained in step S400 are processed using bird's-eye view encoding, a CenterPoint detector head, and non-maximum suppression to obtain the final 3D target detection result. Step S500, which involves performing target detection on the fused features to obtain the target detection result, includes:

[0149] Step 510: Encode the fused features using a bird's-eye view to obtain the features to be identified in the bird's-eye view. More specifically, for the fused feature F... f Further encoding is used to capture important information that is highlighted in a bird's-eye view, such as size, orientation, and speed; more specifically:

[0150] A small 3-layer encoding network is built using basic ResNet modules. Each layer consists of 2 basic modules, and the encoded features are halved in size and doubled in number of channels compared to the original features.

[0151] In addition to the first layer compressing the number of channels from 128 to 64 and then doubling it back to 128; the output feature maps of the first and third layers are extracted, the feature map of the third layer is upsampled to 4 times through bilinear interpolation, and then spliced ​​with the output of the first layer along the feature channels;

[0152] The concatenated features are encoded using two convolutional layers, then an additional upsampling is performed to double the size, followed by two more convolutional layers for encoding. This is to restore the final feature map to the same size as F in step 400. f The same size, thus preserving ample spatial detail;

[0153] Step 520: Perform target detection on the features from step 510 and obtain the target detection results; more specifically, use the CenterPoint detection head to detect targets from the features from step 510; even more specifically:

[0154] For the 10 object detection tasks in the nuScenes dataset, they are divided into six subclasses based on the similarity of size and shape: {car}, {truck, construction_vehicle}, {bus, trailer}, {barrier}, {motorcycle, bicycle}, and {pedestrian, traffic_cone}. Each subclass shares a single task detection head, and all subclasses share a single convolutional encoder. The 3D detection box for each predicted object is B = (x, y, z, l, w, h, θ, v). x v y This includes the target's spatial position relative to the vehicle, its length, width, height, azimuth, and horizontal speed.

[0155] First, the features in step 510 are processed using a convolutional encoder;

[0156] Then, each subclass applies the task detection head to the processed features to obtain the feature heatmap and 3D detection boxes for the subclass. To allow for detection errors and improve the detection rate, the center limit of detection box B is 61.2m on one side along the vehicle body and perpendicular to the vehicle body, the filtering threshold is 0.1, and the maximum number of detection boxes is 500.

[0157] Step 530: Optimize the detection results using non-maximum suppression to obtain the final detection results. More specifically, for cases where a single target instance may have multiple detection boxes, the size-non-maximum suppression method is used to optimize the detection results. More specifically, the suppression method type for the 5 subclasses is "rotate", and the suppression method type for the {barrier} subclass is "circle"; the suppression threshold for the 5 subclasses is 0.2, and the suppression threshold for the {pedestrian,traffic_cone} subclass is 0.5; following the order listed in Step 6.2, the size scaling factors for the 10 target classes are {1.0,0.7,0.7,0.4,0.55,1.1,1.0,1.0,4.5,9.0}; the maximum number of detection boxes after suppression is 83. The optimized detection boxes are the final detection results.

[0158] This application provides a high-precision map alignment and fusion method for multi-view object detection. It utilizes a multi-view object detection baseline to transform image features into a bird's-eye view, and employs high-precision map encoding to semantically approximate map features. Furthermore, it leverages channel and spatial attention for cross-modal feature fusion, thereby mitigating the differences in viewpoint, semantics, and amplitude between the image and map, and ultimately integrating high-precision maps into multi-view 3D object detection. The high-precision map alignment and fusion component of this method has low computational cost, improves 3D object detection accuracy, and promotes autonomous driving perception through the integration of high-precision maps.

[0159] Secondly, based on the same inventive concept, please refer to... Figure 5 This application provides a high-precision map alignment and fusion system suitable for multi-view target detection, including an image feature acquisition module 100, a map feature acquisition module 200, a cross-modal feature acquisition module 300, a fusion feature acquisition module 400, and a target detection module 500. The image feature acquisition module 100 acquires image features of a multi-view image to be detected at a given time. The map feature acquisition module 200 acquires map features of a high-precision map within the vehicle's perception range, recording the vehicle's position and direction at a given time. The cross-modal feature acquisition module 300 is communicatively connected to the image feature acquisition module 100 and the map feature acquisition module 200, and is used to stitch together image features and map features to acquire cross-modal features. The fusion feature acquisition module 400 is communicatively connected to the cross-modal feature acquisition module 300, and is used to perform amplitude adjustment and map information enhancement processing on the cross-modal features to acquire fused features. The target detection module 500 is communicatively connected to the fusion feature acquisition module 400, and is used to perform target detection on the fused features to acquire target detection results. Thirdly, embodiments of this application also provide a readable storage medium.

[0160] This application stores a high-precision map alignment and fusion program suitable for multi-view target detection on a readable storage medium, wherein when the high-precision map alignment and fusion program suitable for multi-view target detection is executed by a processor, it implements the steps of the high-precision map alignment and fusion method suitable for multi-view target detection as described above.

[0161] The method implemented when the high-precision map alignment and fusion program for multi-view target detection is executed can be referred to in the various embodiments of the high-precision map alignment and fusion method for multi-view target detection in this application, and will not be repeated here.

[0162] It should be noted that the sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0163] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device to execute the methods described in the various embodiments of this application.

[0164] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A high-precision map alignment and fusion method suitable for multi-view target detection, characterized in that, Includes the following steps: Obtain image features of a multi-view image to be detected at a given time; The process involves obtaining map features from a high-precision map within the vehicle's perception range, based on the vehicle's position and direction recorded at a given time. This includes cropping a local high-precision map within the vehicle's perception range from the high-precision map based on the vehicle's position and direction recorded at the given time; obtaining a local map representation of the local high-precision map; obtaining image labeling information from the local map representation; appending image labeling information to the local map representation to obtain a balanced map representation; and performing dilated convolution processing on the balanced map representation to obtain map features. The process of obtaining the local map representation of the local high-precision map includes: rasterizing the local high-precision map into multiple grids; for each semantic layer of the local high-precision map, comparing the memory absence of each grid center within the semantic annotation range of the semantic layer layer, and using the binary image containing the determination information of the memory absence of each semantic layer layer as the local map representation. The process of obtaining image labeling information for the local map representation includes: constructing a new binary image; obtaining the position information of all semantic layer images in the local map representation; if an image in the local map representation and the new binary image are at the same position, then the image labeling information of that image in the new binary image is marked as a first label; if an image in the local map representation and the new binary image are at different positions, then the image labeling information of that image in the new binary image is marked as a second label. By stitching together image features and map features, cross-modal features can be obtained; Amplitude adjustment and map information enhancement processing are performed on cross-modal features to obtain fused features; Target detection is performed on the fused features to obtain the target detection results.

2. The high-precision map alignment and fusion method for multi-view target detection as described in claim 1, characterized in that, Adding image labeling information to the local map representation to obtain a balanced map representation includes: By stitching together the channels of the binary images of each semantic layer in the local map representation and a new binary image containing image labeling information, a balanced map representation is obtained.

3. The high-precision map alignment and fusion method for multi-view target detection as described in claim 1, characterized in that, The dilated convolution process for balanced map representation includes map features such as: For the balanced map representation, the first map extraction features are obtained through dilated convolutional layers, SE modules, and downsampling convolutional layers. The feature dimension of the dilated convolutional layer is the first dilated convolution feature dimension, and the feature dimension of the downsampling convolutional layer is the first downsampling convolution feature dimension. For the first map feature extraction, the second map feature extraction is obtained through dilated convolutional layer, SE module, and downsampling convolutional layer. The feature dimension of the dilated convolutional layer is the second dilated convolution feature dimension, and the feature dimension of the downsampling convolutional layer is the second downsampling convolution feature dimension. For the second map feature extraction, the third map feature extraction is obtained through dilated convolutional layer, SE module, and local convolutional layer. The feature dimension of the dilated convolutional layer is the same as the feature dimension of the second dilated convolution, and the feature dimension of the local convolutional layer is the same as the feature dimension of the second dilated convolution, thus obtaining the third map feature extraction. The features extracted from the second map and the features extracted from the third map are added element by element to obtain the final map features; Among them, the first dilated convolution feature dimension < the second dilated convolution feature dimension < the third dilated convolution feature dimension, and the first downsampled convolution feature dimension < the second downsampled convolution feature dimension.

4. The high-precision map alignment and fusion method for multi-view target detection as described in claim 1, characterized in that, The process of adjusting the amplitude of cross-modal features and enhancing map information to obtain fused features includes: Amplitude adjustment is performed on cross-modal features to obtain channel attention; Map information enhancement processing is applied to cross-modal features to obtain spatial attention; The channel attention and spatial attention are multiplied and then normalized using sigmoid to obtain the fused features.

5. The high-precision map alignment and fusion method for multi-view target detection as described in claim 4, characterized in that, The amplitude adjustment of cross-modal features and acquisition of channel attention includes: Global average pooling is applied to cross-modal features, and the features are flattened to obtain feature vectors. The feature vector is compressed by a fully connected layer and then normalized and ReLU activated to obtain the compressed feature vector. Channel attention is obtained by restoring the number of channels in the compressed feature vector through a fully connected layer.

6. The high-precision map alignment and fusion method for multi-view target detection as described in claim 4, characterized in that, The step of performing map information enhancement processing on cross-modal features to obtain spatial attention includes: For cross-modal features, convolution compresses the feature channels and then performs normalization and ReLU activation to obtain the first feature; For the first feature, dilated convolution is performed to extract the feature, and then normalization and ReLU activation are performed to obtain the second feature; For the second feature, dilated convolution is performed to extract the feature, and then normalization and ReLU activation are performed to obtain the third feature; Compress the third special channel to eliminate channel dimension and obtain spatial attention.

7. A high-precision map alignment and fusion system suitable for multi-view target detection, characterized in that, This method is applicable to performing the high-precision map alignment and fusion method for multi-view target detection as described in any one of claims 1-6.

Images