A bev perception method fusing global scene and local instance features

Abstract

This invention proposes a BEV perception method that integrates global scene and local instance features. It constructs a two-layer feature extraction module comprising a global scene feature extraction unit and a local instance feature extraction unit. The global scene feature extraction unit extracts a 2D top-down representation covering the entire scene from multi-view image sequences while preserving the scene's spatial structure information. The local instance feature extraction unit extracts fine-grained features of individual targets from the images. The local instance features and global scene features are weighted and combined, and spatial alignment is used to generate an instance-level top-down feature representation. 3D target recognition is performed on the instance-level top-down feature representation to obtain the probability of belonging to different target categories, and the 3D position information, size, orientation, and motion parameters of each target are output. This invention's BEV perception method, integrating global scene and local instance features, achieves comprehensive optimization of 3D target detection, significantly improving detection accuracy, robustness, and temporal stability.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and autonomous driving, and in particular to a BEV perception method that integrates global scene and local instance features. Background Technology

[0002] 3D object detection from multi-view images is a key technology in fields such as autonomous driving, intelligent transportation, and robot navigation. Bird's Eye View (BEV) perception methods can provide regularized and uniform-scale environmental perception by converting image information from multiple perspectives into a unified top-down representation. This makes the algorithm easier to handle multi-view and multi-sensor inputs, improving the stability and accuracy of perception. Therefore, BEV 3D object detection methods based on multi-camera images have significant application value in fields such as autonomous driving.

[0003] Existing BEV perception methods typically employ the following technical approach: First, feature extraction is performed on multi-view images using a shared encoder (such as ResNet, VGG, etc.) to obtain multi-scale feature maps; second, a depth estimation unit predicts the depth probability distribution for each pixel, mapping the 2D image features to 3D space; third, coordinate transformation and feature pooling are used to convert the 3D features into a top-view feature representation; finally, the detection network outputs the 3D position, size, and orientation information of the target. While these methods have achieved some progress in detection accuracy, they suffer from the following technical problems: 1. Lack of instance-level details in global features: Existing methods mainly rely on global scene feature extraction, generating top-view features through depth estimation and viewpoint transformation. However, these global features often lack details specific to individual target instances. 1. Fine-grained modeling leads to insufficient accuracy in detecting target boundaries, small targets, and distant targets; 2. Severe depth ambiguity: In complex scenes, multiple targets may be distributed at different depths along the same light path. Existing methods lack an effective mechanism to distinguish targets at different depths, making it easy to misidentify or miss occluded or overlapping targets; 3. Significant background noise interference: When extracting target features, existing methods often include background features in the calculation, resulting in significant interference from background noise on target features, especially prone to false detections in complex background scenes; 4. Insufficient temporal fusion strategy: Although existing methods introduce temporal modeling, they mainly rely on simple feature splicing or attention matching, lacking an adaptive fusion strategy for historical frame information, resulting in insufficient temporal stability.

[0004] To address these issues, some studies have attempted to improve BEV perception performance by introducing instance-level feature enhancement or attention mechanisms. However, these improvements still have the following limitations: First, the lack of an effective mechanism for separating the foreground and background results in instance features still containing significant background noise, making it difficult to accurately extract fine-grained features of the target. Second, they fail to fully utilize the spatial angular relationships between target instances, making it difficult to effectively distinguish targets at different depths along the same line of sight, especially in occluded scenes. Third, the lack of an effective fusion strategy for global scene features and local instance features prevents the full utilization of their complementarity, making it impossible to simultaneously maintain global perception capability and local detail accuracy. Therefore, how to construct a two-layer feature extraction module that, while maintaining global scene perception capability, enhances fine-grained modeling of individual target instances through foreground feature selection and angle-adjustment attention mechanisms, and effectively fuses global and local features through adaptive feature fusion strategies, has become a key problem that current BEV 3D target detection technology urgently needs to solve. Summary of the Invention

[0005] To address the technical problems of insufficient global feature space integrity and difficulty in identifying occluded targets in traditional BEV perception methods, this invention proposes a BEV perception method that integrates global scene and local instance features. By constructing a two-layer feature extraction module, local instance features are enhanced on the basis of global top-down features. Foreground feature filtering and angle-adjusted attention mechanisms are used to effectively distinguish targets at different depths. Finally, an adaptive feature fusion strategy is used to organically combine global and local features, thereby significantly improving the accuracy and robustness of 3D target detection.

[0006] To achieve the above objectives, the technical solution of the present invention is implemented as follows: a BEV perception method that integrates global scene and local instance features, comprising the following steps:

[0007] S1. Construct a two-layer feature extraction module, including a global scene feature extraction unit and a local instance feature extraction unit. The global scene feature extraction unit extracts a two-dimensional top-down representation covering the entire scene from the multi-view image sequence and retains the spatial structure information of the scene to obtain global scene features; the local instance feature extraction unit obtains fine-grained features of individual targets from the image to obtain local instance features.

[0008] S2, Feature Fusion: Weighted combination of local instance features and global scene features, and high-density instance-level top-down feature representation is generated through spatial alignment operation;

[0009] S3. Perform 3D target recognition on the instance-level top-view feature representation, obtain the probability of belonging to different target categories, and output the 3D position information, size, orientation and motion parameters of each target.

[0010] Preferably, the global scene feature extraction unit includes a multi-view image encoding unit and a depth estimation unit. The multi-view image encoding unit processes the input multi-view image sequence through a shared encoder to extract multi-scale features of each frame to preserve the spatial structure information and texture details of the image. The multi-view image encoding unit supports cross-view feature consistency and captures local and global information through convolution or Transformer modules.

[0011] The depth estimation unit establishes a mapping relationship between the multi-scale features obtained by the multi-view image coding unit and the three-dimensional geometric space to predict the depth probability distribution of each pixel.

[0012] By combining multi-scale features with depth probability distribution, and using outer product operation or depth weighted mapping to perform dimensionality upscaling operation, pseudo-3D point cloud features are generated.

[0013] A spatial mapping unit is established, and the pseudo-3D point cloud is mapped to the sensor coordinate system or vehicle coordinate system through camera intrinsic parameters and coordinate transformation. The 3D features are projected onto the BEV plane through top-view feature pooling operation to generate a preliminary top-view feature representation.

[0014] The preliminary top-down feature representation generated in the previous T-1 frames is spatiotemporally aligned and fused to the preliminary top-down feature representation of the current frame T through a temporal encoder to generate global top-down features. Global average pooling is then performed on the global top-down features to extract the overall semantic representation, resulting in global scene features containing scene structure and dynamic information.

[0015] Preferably, during convolution, small-sized convolutional kernels are used to slide within the local receptive field to extract detailed features of edges and textures. As the number of network layers increases, the receptive field expands layer by layer, gradually aggregating a wider range of contextual information to achieve modeling from local to global. During Transformer, images in the multi-view image sequence are divided into patches and flattened into a sequence. The self-attention mechanism is used to calculate the correlation between any positions, so that each position can directly interact with all positions globally, thereby modeling long-distance dependencies at once and capturing global information. At the same time, multi-head attention and hierarchical structure are combined to take into account both local details and global semantic information.

[0016] The shared encoder is a shared deep convolutional neural network encoder that extracts features from multi-view images to obtain image feature representations with multi-scale hierarchical structures. The ResNet101 network is used as the shared deep convolutional neural network encoder, which contains 5 residual blocks, each of which contains multiple convolutional layers and skip connections.

[0017] The depth estimation unit adopts a fully convolutional neural network, which includes an encoder-decoder architecture. The encoder extracts features through multi-layer convolution and pooling operations, and the decoder restores spatial resolution through deconvolution and skip connections. Multi-scale features are mapped to D depth channels through 1×1 convolution, and the depth probability distribution of each pixel is obtained through softmax normalization.

[0018] The dimensionality upscaling operation employs an outer product operation, where the feature vector f(u,v) of each pixel (u, v) is expanded along the depth direction d to generate pseudo-3D point cloud features. Where ⊗ represents the outer product operation, Let be the probability value of the depth probability distribution of pixel (u, v) at depth direction d;

[0019] The spatial mapping unit performs coordinate transformation using camera calibration parameters, including intrinsic parameter matrix K and extrinsic parameter matrix;

[0020] The top-down feature pooling involves placing 3D points into corresponding BEV mesh cells according to their (x, y) coordinates, in the height dimension. The aggregation operation is performed on the top to compress all three-dimensional features within the same BEV grid into a two-dimensional feature vector, forming a BEV feature map, which is the preliminary top-view feature representation.

[0021] The temporal encoder adopts a Transformer architecture, which includes 8 encoder layers. Each layer includes a multi-head self-attention mechanism, a feedforward network, and residual connections. The temporal feature fusion unit performs spatiotemporal alignment through a motion compensation mechanism, spatially deforms the top-view feature representation of historical frames to align it with the top-view feature representation of the current frame, and generates the final global top-view feature.

[0022] Preferably, the local instance feature extraction unit includes a foreground feature filtering unit and an angle-adjusted attention unit. The foreground feature filtering unit selects effective pixel features based on the candidate target region to suppress background interference and enhance the feature expression capability of the target region. The angle-adjusted attention unit dynamically adjusts the feature weights based on the spatial angle relationship between different target instances to distinguish targets at different depths along the same light path, thereby improving the ability to identify occluded targets and dense targets and obtaining local instance features.

[0023] Preferably, the implementation steps of the foreground feature filtering unit are as follows:

[0024] The original image is input into a convolutional neural network to obtain a two-dimensional feature map. The corresponding pixel is selected on the image plane of the two-dimensional feature map from multiple perspectives using the three-dimensional reference coordinates of each target instance, and image features are sampled from the corresponding pixel position.

[0025] In the obtained image features, foreground pixels are filtered, and only foreground pixel features located within the candidate target region are retained to generate high-density instance-level top-down features;

[0026] Background pixels are masked for pixel features that do not belong to the target region to avoid background interference affecting instance features.

[0027] Preferably, the candidate target region is a candidate bounding box predicted by the target detection head; the target bounding box output by the detection network corresponds to an independent target instance.

[0028] The 3D center or 3D bounding box corners of the target instance are projected onto the image plane of each viewpoint through the camera extrinsic and intrinsic parameter matrices to obtain pixel coordinates. A sampling window is established within a set radius with the pixel coordinates as the center, and features at the corresponding positions are sampled on the image feature map through bilinear interpolation.

[0029] The filtering process that retains only foreground pixel features located within the candidate target region is achieved through candidate region mask or ROI bounding box restriction. By using the two-dimensional bounding box or foreground segmentation mask output by the detection network, a binary mask is constructed on the feature map, retaining only pixel features located within the ROI range.

[0030] By determining whether the pixel coordinates fall within the 2D bounding box or instance segmentation mask of any target instance, if they are not within any target area, they are determined to be background areas.

[0031] The foreground pixel selection is performed through two-dimensional bounding box constraints: for each target instance, a two-dimensional bounding box is obtained based on the projection of the three-dimensional bounding box onto the image plane, only the pixel features located within the two-dimensional bounding box are retained, and pixels that do not belong to the target area are filtered out to generate high-density instance-level top-down features.

[0032] The background pixel masking is achieved by setting a masking mechanism: for each pixel position, if it is not within the bounding box of any target instance, its feature value is multiplied by a weight decay factor of 0.1.

[0033] Preferably, the implementation steps of the angle-adjustable attention unit are as follows:

[0034] The angular deviation on the BEV plane is calculated based on the center coordinates in the three-dimensional space of each pair of target instances;

[0035] In the self-attention mechanism, the attention weights between each target instance are dynamically adjusted using angle deviation;

[0036] The attention weights after angle adjustment are applied to the instance-level top-down features, which enable the instance-level top-down features to have a stronger ability to identify target instances at different depths, generating depth-discriminating features, which are local instance features.

[0037] Preferably, the method for dynamically adjusting the attention weight is as follows: for instances belonging to the same target instance or spatially adjacent instances, the attention connection between feature vectors is enhanced, that is, the similarity weight in self-attention calculation is used to make the related instance-level top-down features mutually reinforce information transmission; for instances located at different depths or on different light paths, the attention weight is reduced to suppress feature interference.

[0038] A multi-head attention structure is adopted, which sets up multiple parallel attention heads to model the correlation between target instances in different feature subspaces, and adjusts the attention weights by combining the angle difference between target instances;

[0039] The target instance predicted by the 3D target detection network or BEV detection head includes 3D center coordinates, length, width, height, and orientation information; for instance i and instance j, the 3D center coordinates are respectively... and Calculate the azimuth angle relative to the sensor origin: , Angular deviation is The value range is [-π, π]. It is the arctangent function in the four quadrants;

[0040] The attention weight adjustment is achieved through an angle control function. The implementation is as follows: where σ is the angular bandwidth parameter, and the final attention weights are calculated as follows: ,in This is a correlation function based on feature similarity. This is used to normalize the attention weights, ensuring that the sum of the weights for the same instance query and all keys is 1. For the query characteristics of instance i, Here are the key features of instance i, used for similarity matching with the query; the final output features of instance i are: ;in, Represents the value characteristics of the target instance;

[0041] The depth differentiation feature is achieved by enhancing the feature differences between targets at different depths along the same angular direction. It classifies the angle of the instance center on the BEV plane, identifying multiple target instances along the same angular direction, and then combines this with depth... Coordinates are used for depth differentiation; for multiple targets in the same angle direction, a depth-weighted mechanism is used to distinguish them:

[0042] ;

[0043] in, For depth weight parameters, and These are the mean and standard deviation of the depth, respectively. The three-dimensional center depth of the target instance. This is an instance-level top-down feature after angle adjustment. This ultimately enhances the instance-level top-down features, which provide greater depth discrimination capabilities.

[0044] Preferably, the feature fusion step is as follows:

[0045] For each target instance, extract instance-level top-down features at different resolutions to generate multi-scale instance features with high and low resolutions;

[0046] Spatiotemporally align multi-scale instance features with instance-level top-down features of historical frames from the previous T-1 frames;

[0047] An adaptive feature fusion strategy is used to perform weighted fusion of multi-scale instance features and instance-level top-down features of historical frames.

[0048] Preferably, the multi-scale instance feature extraction is achieved through a feature pyramid network, which includes four scale levels;

[0049] The spatiotemporal alignment uses motion compensation or optical flow estimation to correct the displacement of the target in consecutive frames, and maps the instance-level top-down features of historical frames to the reference coordinate system of the current frame through coordinate transformation.

[0050] The adaptive feature fusion strategy adopts an adaptive fusion method based on attention or weighting strategy, which weights and combines the multi-scale instance features of the current frame with the instance-level top-down features aligned with historical frames to generate a high-density, information-rich instance-level top-down feature table.

[0051] The adaptive feature fusion strategy is implemented through a learnable fusion weight network. The fusion weight network adopts a fully convolutional neural network structure. The input is the concatenation of multi-scale instance features and historical top-down features of historical frames, and the output is the fusion weight.

[0052] The 3D target recognition predicts the target category probability for each grid or anchor point of the input instance-level top-down features using a fully convolutional neural network, Transformer structure, or multi-head attention mechanism; and obtains the confidence score of each category through softmax to quantify the probability that the grid / anchor point belongs to different target categories.

[0053] A fully convolutional neural network contains multiple convolutional layers and the ReLU activation function;

[0054] The bounding box prediction network outputs 3D bounding box parameters for each target, including the target center position, target size, orientation angle, and velocity vector.

[0055] By utilizing instance-level top-down features aligned with historical frames and a motion information encoding module, the dynamic behavior of the target is captured, ensuring accurate 3D positioning and motion description of both static and dynamic targets. Historical frame features are input through the BEV features of the previous T-1 frames obtained by the temporal feature fusion unit in step S204. The motion information encoding module extracts dynamic information such as the target's displacement and velocity trends, which is then fused with the top-down features of the current frame to model the target's dynamic behavior.

[0056] The bounding box prediction network is based on a fully convolutional network and predicts the target center, target size, and target orientation angle in the BEV plane for each grid or anchor point.

[0057] By using a motion information prediction network to predict the velocity vector for each grid and combining it with instance-level top-down features from historical frames to capture the target's motion trend, dynamic target velocity estimation is achieved.

[0058] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0059] 1. This invention achieves the organic fusion of global scene features and local instance features by constructing a two-layer feature extraction module. Compared with the traditional BEV perception method that only relies on global features, this invention significantly enhances the fine-grained modeling capability of individual target instances while maintaining the global scene perception capability. In particular, the accuracy is significantly improved in the detection of target boundaries, small targets and distant targets, effectively solving the problem of insufficient global feature space integrity in existing methods.

[0060] 2. This invention innovatively introduces a foreground feature screening unit. Through mechanisms such as three-dimensional query point sampling, foreground pixel screening, and background pixel masking, it effectively achieves the separation of foreground and background, and significantly reduces the interference of background noise on target features. Compared with existing methods, this invention significantly reduces the false detection and false negative rates in complex backgrounds and scenes with large changes in lighting, thereby improving the robustness of detection.

[0061] 3. This invention utilizes the spatial angular relationship between target instances by adjusting the attention unit by angle. It adopts a multi-head attention structure and combines the angle difference for dynamic weight adjustment, which effectively solves the problem of distinguishing targets at different depths along the same light path. Compared with existing methods, this invention significantly improves the detection accuracy in occluded and overlapping scenes, especially the detection capability of occluded targets is significantly improved.

[0062] 4. This invention employs an adaptive feature fusion strategy and a learnable fusion weight network to dynamically generate fusion weights based on the statistical characteristics or semantic information of the input features. This achieves complementary advantages between local instance features and global scene features. Compared with traditional simple feature concatenation or fixed weight fusion methods, the adaptive fusion strategy of this invention can better adapt to the feature characteristics of different scenes and different targets, thereby improving the fusion effect and detection accuracy.

[0063] 5. This invention uses a multi-head attention structure combined with an angle control function to compute the correlation between target instances in parallel in multiple feature subspaces. Compared with the single-head attention mechanism, multi-head attention can capture the complex relationships between features from different angles, and has significant advantages, especially in dealing with occluded and overlapping targets. It significantly improves the model's ability to model complex scenes and its detection accuracy.

[0064] 6. The BEV perception method of the present invention, which integrates global scene and local instance features, achieves comprehensive optimization of 3D target detection through the organic combination of innovative mechanisms such as two-layer feature extraction, foreground feature screening, angle-adjusted attention, and adaptive feature fusion. Compared with existing BEV perception methods, the present invention has achieved significant improvements in detection accuracy, robustness, and temporal stability. It is particularly suitable for application scenarios with high perception performance requirements, such as autonomous driving, intelligent transportation, and robot navigation, and has important practical value and broad application prospects. Attached Figure Description

[0065] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0066] Figure 1 This is a schematic diagram of the overall architecture of the present invention.

[0067] Figure 2 This is a schematic diagram of the specific architecture of the angle-controlled attention unit used in this invention.

[0068] Figure 3 This is a schematic diagram of the detection results of the model of the present invention. Detailed Implementation

[0069] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0070] like Figure 1 As shown, a BEV perception method that integrates global scene and local instance features constructs a two-layer feature extraction module, including a global scene feature extraction unit and a local instance feature extraction unit. The global scene feature extraction unit generates a two-dimensional top-down representation covering the entire scene from multi-view image sequences while preserving spatial structure information. The local instance feature extraction unit extracts fine-grained features of individual targets from the image to enhance the spatial integrity of the global features. Feature fusion is performed, combining local instance features with global scene features in a weighted manner to generate a high-density top-down feature representation. Based on the fused top-down feature representation, the three-dimensional position information, size, orientation, and motion parameters of each target are output. The local instance feature extraction unit includes a foreground feature selection unit and an angle-adjusted attention unit. The foreground feature selection unit selects effective pixel features based on candidate target regions to suppress background interference, while the angle-adjusted attention unit dynamically adjusts feature weights based on the spatial angular relationship between different target instances to distinguish targets at different depths along the same light path. This invention effectively solves the problems of insufficient global feature space integrity and difficulty in identifying occluded targets in traditional BEV perception methods, improving the accuracy and robustness of 3D target detection. It meets the real-time and accuracy requirements of autonomous driving systems for perceiving complex scenes and can be well deployed and applied in intelligent driving systems. The specific implementation method of this invention includes the following steps:

[0071] S1. Construct a two-layer feature extraction module, including a global scene feature extraction unit and a local instance feature extraction unit. The global scene feature extraction unit is used to extract a two-dimensional top-down representation covering the entire scene from multi-view image sequences to obtain global scene features, while preserving the spatial structure information of the scene. The local instance feature extraction unit is used to obtain fine-grained features of individual targets from the image to obtain local instance features, thereby enhancing the spatial integrity of the global features.

[0072] In step S1, the steps for constructing the global scene feature extraction unit are as follows:

[0073] S201. Construct a multi-view image coding unit to process the input multi-view image sequence. The multi-view image coding unit processes the input multi-view image sequence through a shared encoder, extracting multi-scale features from each frame to preserve the spatial structure information and texture details of the image. The multi-view image coding unit supports cross-view feature consistency and captures local and global information through convolution or Transformer modules, providing high-quality feature representations for subsequent depth estimation and 3D mapping. When using convolution, the network uses small-sized convolutional kernels (such as 3×3) to slide within the local receptive field to extract detailed features such as edges and textures. As the network depth increases, the receptive field expands layer by layer, thereby gradually aggregating a larger range of contextual information and achieving modeling from local to global. When using Transformer, the image is divided into patches and flattened into a sequence. The self-attention mechanism is used to calculate the correlation between arbitrary positions, allowing each position to directly interact with all positions globally, thereby modeling long-distance dependencies at once and capturing global information. At the same time, the combination of multi-head attention and hierarchical structure takes into account both local details and global semantic information. The input dimensions are [B, T, N, 3, H1, W1], where B is the batch size (4), T is the number of time frames (8), N is the number of cameras (6), 3 represents the number of channels, and H1 and W1 are the image height and width (900×1600), respectively. The shared encoder is a shared deep convolutional neural network encoder used for feature extraction from multi-view images to obtain image feature representations with multi-scale hierarchical structures, i.e., global scene features. This provides high-level semantic and spatial structural information for subsequent top-down feature construction and fusion. A ResNet101 network is used as the shared encoder to extract multi-scale features from images at different viewpoints. The ResNet101 network contains 5 residual blocks, each containing multiple convolutional layers and skip connections to improve the perception of targets at different viewpoints and scales.

[0074] S202. Construct a depth estimation unit to predict the depth probability distribution of each pixel, forming a dense depth map or depth probability volume. The depth estimation unit establishes a mapping relationship between the two-dimensional image features (multi-scale features obtained by the multi-view image encoding unit) obtained from the image through a convolutional neural network and the three-dimensional geometric space, so that each pixel feature (each pixel of the image) corresponds to a position in three-dimensional space, thus providing a foundation for subsequent pseudo-3D point cloud feature generation. The depth estimation unit adopts a fully convolutional neural network, including an encoder-decoder architecture. The encoder extracts features through multiple layers of convolution and pooling operations, and the decoder restores spatial resolution through deconvolution and skip connections. The output depth probability distribution has dimensions [B, T, N, D, H, W], where D is the number of depth intervals, set to 64, the depth range is set to [0.5m, 100m], and m is meters. The depth probability distribution of each pixel is obtained through softmax normalization. The fully convolutional neural network maps the fused multi-scale features to D depth channels through 1×1 convolution and then normalizes them through softmax, thus outputting the corresponding depth probability distribution for each pixel.

[0075] S203. Perform a 3D feature upscaling operation. This involves combining 2D image features (multi-scale features) with depth information (the depth probability distribution obtained in step S202) and generating pseudo-3D point cloud features using an outer product operation or depth-weighted mapping. This step expands the 2D information of each frame into 3D space, preserving the structural information of the global scene while enhancing the ability to represent targets at different heights or occluded locations. The upscaling operation uses an outer product operation. For each pixel (u, v), its feature vector f(u, v) is expanded along the depth direction d to generate pseudo-3D point cloud features. Where ⊗ represents the outer product operation, Let be the probability value of pixel (u, v) at the depth direction d. The pseudo-3D point cloud is generated by convolving the features of a 2D image with a depth estimation unit, outputting D depth channels, and then normalizing the depth probability distribution of each pixel using the softmax function. The generated pseudo-3D point cloud feature dimensions are [B, T, N, C, D, H, W], where C is the number of feature channels, set to 256.

[0076] S204. Establish a spatial mapping unit to map the pseudo-3D point cloud to the sensor coordinate system or vehicle coordinate system using camera intrinsic parameters and coordinate transformation. The spatial mapping unit performs coordinate transformation using camera calibration parameters (intrinsic parameter matrix K, extrinsic parameter matrix [R|t]). For each pixel (u, v) and depth direction d, its 3D coordinates in the camera coordinate system are calculated using the back projection formula: Then, the coordinates are transformed to the sensor coordinate system using the extrinsic parameter matrix: Finally, the 3D features are projected onto the BEV plane (2D top-view plane) through top-view feature pooling, generating a preliminary top-view feature representation with dimensions [B, T, C, H2, W2], where H2=200 and W2=200. This process ensures the consistency of spatial structure and scene geometry information on the top-view plane. 3D features are derived from 2D pixel features. Its corresponding depth probability Three-dimensional point cloud features formed by expanding the outer product along the depth dimension Then, based on the 3D coordinates (x, y, z) obtained from back projection, corresponding spatial positions are assigned, thus obtaining pseudo-3D point cloud features with spatial coordinates. Top-view feature pooling places 3D points into corresponding BEV mesh cells according to their (x, y) coordinates, in the height dimension... Aggregation operations (such as sum / mean / max pooling or weighted summation) are performed on the grid to compress all three-dimensional features within the same BEV raster into a single two-dimensional feature vector, ultimately forming a vector of size . BEV feature map.

[0077] S205, the temporal feature fusion unit performs spatiotemporal alignment on the preliminary top-view feature representation generated in the previous T-1 frames, and fuses it with the preliminary top-view feature representation of the current frame T through a temporal encoder to generate the final global top-view feature. The temporal encoder adopts a Transformer architecture, containing 8 encoder layers, each layer including a multi-head self-attention mechanism (8 heads), a feedforward network, and residual connections. The temporal feature fusion unit performs spatiotemporal alignment through a motion compensation mechanism, spatially deforming the features (top-view feature representation) of historical frames to align with the features of the current frame, generating the final global top-view feature with dimensions [B, C, H2, W]. The global top-view feature can capture dynamic scene information and target motion patterns, providing high-quality input for subsequent local instance feature enhancement and 3D target recognition. By utilizing the vehicle pose change (pose transformation matrix provided by IMU / GNSS), the BEV features of historical frames are mapped to the coordinate system of the current frame through rigid body transformation, achieving spatial alignment. The features of historical frames are the previous... The size of the frame after spatial mapping and BEV projection is The BEV feature map is generated and cached frame by frame by the multi-view image encoding, depth estimation, and spatial mapping unit. The current frame feature is the top-view feature map obtained by multi-view encoding, depth estimation, and BEV projection of the current frame T, and is used as the reference coordinate system for temporal fusion.

[0078] By performing global average pooling on the fused global top-down features, the overall semantic representation is extracted, thereby obtaining a global scene feature vector containing scene structure and dynamic information.

[0079] The local instance feature extraction unit includes a foreground feature selection unit and an angle-adjusted attention unit. The foreground feature selection unit selects effective pixel features based on candidate target regions to suppress background interference and enhance the feature representation of the target region; candidate target regions are candidate boxes predicted by the target detection head. The angle-adjusted attention unit dynamically adjusts feature weights based on the spatial angular relationships between different target instances to distinguish targets at different depths along the same light path, thereby improving the recognition ability of occluded and dense targets. Different target instances are target bounding boxes output by the detection network, with each bounding box corresponding to an independent target instance.

[0080] In step S1, the steps for constructing the foreground feature filtering unit of the local instance feature extraction unit are as follows:

[0081] S301. Based on the 3D query point sampling image features, select the corresponding pixel on the multi-view image plane using the 3D reference coordinates of each target instance, and sample image features from the corresponding pixel position. The 3D center or 3D bounding box corners of the target instance are then sampled using camera extrinsic parameters. and internal reference Pixel coordinates are obtained by projecting the image onto the image plane at each viewpoint. Then, a sampling window is established within a set radius (e.g., 50 pixels) centered on the pixel coordinates, and features at the corresponding positions are sampled on the image feature map using bilinear interpolation. The 3D query point is projected onto the image plane of each viewpoint using the camera intrinsic matrix K and extrinsic matrix [R|t] by projecting the 3D center coordinates (x, y, z) of the target instance onto the image plane of each viewpoint. The pixel coordinates (u, v) on the image plane are obtained, and the sampling radius is set to 50 pixels to determine the range of pixel locations to be sampled. This step ensures that the local features of each target instance can accurately correspond to the effective pixel locations in the two-dimensional image, while maintaining the accuracy of the three-dimensional spatial information.

[0082] S302. Foreground pixel selection: From the sampled image features, only foreground pixel features located within the candidate target region are retained to generate high-density instance-level top-down features. Image features are obtained by inputting the original image into a convolutional neural network to obtain a two-dimensional feature map, and then projecting the image onto this feature map. The location is obtained through indexing or interpolation. This filtering process is achieved through candidate region masks or ROI bounding boxes. By detecting the two-dimensional bounding boxes or foreground segmentation masks output by the network, a binary mask is constructed on the feature map, retaining only pixel features within the ROI range. This effectively enhances the feature representation capability of the target instance, making local features more concentrated and richer in target information.

[0083] Foreground pixel selection is performed using 2D bounding box constraints. For each target instance, the 2D bounding box is obtained by projecting its 3D bounding box onto the image plane. Only pixel features within the 2D bounding box are retained, while pixels not belonging to the target region are filtered out, generating high-density instance-level top-down features with dimensions of [dimension not specified]. ,in, For the target number of instances, and Spatial resolution of instance features. These represent the top-left and bottom-right pixel coordinates of the two-dimensional bounding box on the image plane, respectively. The x-coordinate of the left boundary of the bounding box. The ordinate of the upper boundary of the bounding box. The x-coordinate of the right boundary of the bounding box. This represents the ordinate of the lower boundary of the bounding box. The 3D bounding box is typically obtained by a 3D object detection network regressing on BEV features, predicting the center coordinates of the target. Length, width and height and orientation angle This allows for the construction of a complete 3D bounding box.

[0084] S303. Background pixel masking: Pixel features that do not belong to the target area are masked or ignored to avoid background interference affecting instance features. This is achieved by determining pixel coordinates. Whether the region falls within the 2D bounding box or instance segmentation mask of any target is considered a background region. If it does not fall within any target region, it is determined to be a background region. By masking background pixels, interference from environmental background, other targets, or noise pixels can be suppressed, improving the accuracy and recognition capability of instance-level top-down features, and providing reliable input for subsequent local instance feature fusion and 3D target recognition.

[0085] Background pixel masking is achieved by setting a masking mechanism. For each pixel location, if it is not within the bounding box of any target instance, its feature value is multiplied by a weight decay factor of 0.1 to suppress the contribution of the background region to instance-level top-down feature extraction. The background masking formula is as follows:

[0086]

[0087] in, It is a boolean conditional variable representing pixels. Whether it is located inside the two-dimensional bounding box of any target. These are the pixel feature values ​​after background masking, i.e., the original features. The result after multiplying by the corresponding weights. The target instance is predicted by the preceding 3D target detection network or BEV detection head, including the 3D bounding box of each target and its 2D projection box on the image plane.

[0088] In step S1, the steps for constructing the angle-controlled attention unit of the local instance feature extraction unit are as follows:

[0089] S401. Calculate the center angle difference of the instances. Based on the center coordinates of each pair of target instances in the three-dimensional space, calculate the angle deviation of the instance on the top plane (specifically the BEV plane). This step converts the target position information in the three-dimensional space into a two-dimensional angle relationship by projecting onto the top plane, thereby quantifying the relative spatial orientation between the targets and providing an accurate angle reference for subsequent attention weight adjustment.

[0090] Target instances predicted by a 3D target detection network or BEV detection head, each instance containing 3D center coordinates Dimensions, width, height, and orientation information. For instance i and instance j, their three-dimensional center coordinates are respectively... and Calculate the azimuth angle relative to the sensor origin (set at the center of the vehicle): , Angular deviation is The value range is [-π, π]. It is an operator, specifically a four-quadrant arctangent function.

[0091] S402. Adjusting Attention Weights: In the self-attention mechanism, the attention weights between target instances are dynamically adjusted using angular deviation. Specifically, for instances belonging to the same target instance or spatially adjacent, the attention connection between their feature vectors is enhanced. That is, the similarity weight in the self-attention calculation is used to make related instance features mutually reinforce information transmission. By calculating the Euclidean distance between target instances on the BEV plane, if the distance between instance centers is less than a set threshold, they are considered spatially adjacent. The same instance is directly distinguished by the target ID predicted by the detection network, and adjacent instances are filtered by calculating the center coordinate distance in BEV. For instances located at different depths or on different light paths, their attention weights are reduced to suppress feature interference. By comparing the depth value (z-coordinate) of the instance center or whether the light paths overlap, if the depth difference is large or the light rays projected to the same pixel do not overlap, they are considered to belong to different depths or different light paths. Through this step, the focus enhancement of local instance features can be achieved, while reducing mutual interference between different target instances and improving feature discriminability. Local instance features are obtained by foreground pixel filtering + 3D query point sampling + BEV projection + background masking, resulting in a top-view feature representation of each target instance.

[0092] A multi-head attention structure is employed, which models the correlations between target instances in different feature subspaces by setting multiple parallel attention heads. The attention weights are then adjusted based on the angle differences between instances, thereby enhancing the expressive power and depth discrimination capabilities of instance-level top-down features. The number of attention heads is set to 8, with each attention head having a dimension of 32. Attention weight adjustment is achieved through an angle control function. Where σ is the angular bandwidth parameter, set to 0.5 radians, the final attention weights are calculated as follows: ,in This is a correlation function based on feature similarity. This is used to normalize the attention weights so that the sum of the weights of the same instance query and all keys is 1, thus forming a probability distribution. For instance i, the query feature is used to calculate its relevance to other instance features. Let be the key features of instance i, used for similarity matching with the query. The attention calculation method is expressed as follows:

[0093]

[0094] in, , , These represent the query features, key features, and value features of the target instance, respectively. Represents the correlation function based on feature similarity. Representation of instances With examples The angular difference in the top-down plane This is an angle adjustment function used to dynamically adjust attention weights based on the angle difference. The final output feature of instance i represents the weighted sum of instance-level attention features, which integrates angle adjustment and multi-instance information. exp is an exponential function used to map the feature similarity after angle adjustment to a positive number, and works with softmax to normalize the weights.

[0095] Figure 2 This is a visual representation of all the above formulas.

[0096] S403. Generate depth-discriminating features and apply the angle-adjusted attention weights to the instance-level top-down features, enabling the instance-level top-down features to have a stronger ability to distinguish target instances at different depths. This process can enhance the discrimination effect of targets with depth overlap or occlusion on the top-down feature plane, so that local instance features can still accurately reflect the spatial position and depth information of each target after being fused with global scene features.

[0097] Depth-discriminative features are achieved by enhancing the feature differences between targets at different depths within the same angular direction. This is done by classifying the angles of instance centers on the BEV plane, identifying multiple target instances within the same angular direction, and then combining these features with their depth. Coordinates are used for depth differentiation. For multiple targets in the same angle direction, a depth-weighted mechanism is used to distinguish them:

[0098] ;

[0099] in, The depth weight parameter is set to 0.5. and The mean and standard deviation of the depth represent the depth of all target instances at the same angular direction. Coordinate calculation statistics: = Average depth, = Depth standard deviation, used to standardize the depth bias of each instance to generate depth-weighted features. The three-dimensional center depth (z coordinate) of the target instance. This refers to the instance-level top-down features after angle adjustment (i.e., features adjusted by attention weights). To ultimately enhance the instance-level top-down feature's depth discrimination capability, in the instance-level top-down feature f attention Add depth weighting to the base.

[0100] Local instance features are obtained by taking steps such as 3D query point sampling, foreground pixel filtering, background pixel masking, and BEV projection to obtain a high-density local feature representation of each target instance on the top-view plane.

[0101] S2. Perform feature fusion, which weights and combines local instance features with global scene features. Through feature weighting and spatial alignment operations, a high-density top-down feature representation is generated, so that the fused features simultaneously contain global scene information and fine-grained information about local targets, thereby improving the completeness and distinguishability of the top-down features.

[0102] In step S2, the specific steps of feature fusion are as follows:

[0103] S501. Construct multi-scale instance features. For each target instance, extract instance-level top-down features at different resolutions to generate high-resolution and low-resolution multi-scale instance features. High-resolution features in the multi-scale instance features are used to capture the fine structural information of the target, such as edges, corners, and local textures; low-resolution features in the multi-scale instance features are used to capture the overall shape and global spatial layout information of the target. Through the construction of multi-scale instance features, each instance feature contains both fine-grained details and retains global contextual information, providing a rich representation for subsequent feature fusion.

[0104] Multi-scale feature extraction is achieved through a Feature Pyramid Network (FPN), which includes four scale levels with resolutions of [200×200, 100×100, 50×50, 25×25]. Each scale level has 256 feature channels, which fully captures the local and global information of the target.

[0105] S502. Combining historical frame top-down features, the multi-scale instance features are spatiotemporally aligned and prepared for fusion with the historical top-down features of the previous T-1 frames. During the alignment process, motion compensation or optical flow estimation is used to correct the target's displacement in consecutive frames, and coordinate transformation is used to map the historical top-down features to the reference coordinate system of the current frame. This step ensures the consistency of instance-level top-down features in the temporal dimension, providing an accurate spatial basis for fusing multi-frame information, while enhancing the representation capability of dynamic targets.

[0106] S503, Adaptive Feature Fusion, employs an adaptive feature mixing strategy to weightedly fuse multi-scale instance features with historical frame top-down features. An adaptive fusion method based on attention or weighting strategies is used to weightedly combine the multi-scale instance features of the current frame with historical top-down features aligned with previous frames. During the fusion process, learnable weights are assigned to features at different scales and time steps, effectively integrating high-resolution details and low-resolution global information, while enhancing the ability to distinguish dynamic targets from static background features. The fusion result generates a high-density, information-rich instance-level top-down feature representation, providing high-quality input for final global top-down feature enhancement and 3D target recognition.

[0107] The adaptive feature fusion strategy is implemented through a learnable fusion weight network. This network employs a fully convolutional neural network structure, taking as input a concatenation of multi-scale instance features and historical top-down features from historical frames, and outputting as fusion weights w∈[0,1]. The fusion formula is: , where w is dynamically generated by the fusion weight network based on the statistical properties of the input features. For the multi-scale instance features of the current frame, The top-down features of historical frames are obtained by the spatial mapping unit and the temporal feature fusion unit of step S204, and then mapped to the reference coordinate system of the current frame through spatiotemporal alignment. The final instance feature after fusion is obtained by weighting the current frame instance feature and the historical frame feature by the fusion weight w.

[0108] In step S3, the specific steps for 3D target recognition are as follows:

[0109] S601. Receive fused top-down features. Input the final high-density instance-level top-down feature representation, generated by fusing instance-level and scene-level top-down features, into the 3D target detection unit. The input features contain rich spatial structure information, fine-grained target information, and multi-frame dynamic information, providing a complete perceptual context for 3D target detection. The role of the 3D target detection unit is to identify the 3D position, size, and orientation of each target on the BEV plane based on the fused top-down features. The fused instance-level top-down feature representation has a dimension of... Where C=256, = =200.

[0110] S602, 3D target recognition: Based on the instance-level top-down feature representation of the input, target category information is predicted. A convolutional network, Transformer, or multi-head attention mechanism is used to predict the target category probability for each grid or anchor point. Each grid or anchor point is a discrete location uniformly divided on the BEV plane (e.g., an H2×W2 grid). This step generates a category confidence score for each potential target region and combines spatial location features to improve the recognition ability of small targets and occluded targets. A fully convolutional integral classification network is used to convolve the features of each grid or anchor point, outputting the predicted probability for each category. Then, a softmax function is used to obtain the confidence score for each category, which quantifies the probability that the grid / anchor point belongs to different target categories.

[0111] 3D target recognition is achieved through a classification network, employing a fully convolutional neural network structure containing multiple convolutional layers and a ReLU activation function, with an output dimension of [missing information]. ,in, Set the target category number to 10 (including vehicles, pedestrians, bicycles, etc.).

[0112] S603, Bounding Box and Motion Information Prediction: Outputs the 3D bounding box position, size, orientation, and motion velocity information for each target. Based on the completed category prediction, it outputs the 3D bounding box parameters for each target, including center position coordinates (x, y, z), size information (l, w, h), and orientation angle. and velocity vector This prediction utilizes both historical frame features and a motion information encoding module to capture the target's dynamic behavior, ensuring accurate 3D localization and motion description of both static and dynamic targets. Historical frame features are obtained by inputting the BEV features of the previous T-1 frames from the temporal feature fusion unit in step S204. The motion information encoding module extracts dynamic information such as the target's displacement and velocity trends, which is then fused with the current frame's top-down features to model the target's dynamic behavior.

[0113] The bounding box prediction network output dimension is The output dimension of the motion information prediction network includes the center coordinates (x, y, z), length, width, and height dimensions (l, w, h), and yaw angle θ. Including velocity vector The bounding box prediction network is based on a fully convolutional network. It predicts seven parameters [x, y, z, l, w, h, θ] for each BEV grid or anchor point, and outputs continuous values ​​through convolution to achieve 3D bounding box regression. [x, y, z] are used to locate the target center, [l, w, h] represent the target size, and θ represents the target's orientation angle in the BEV plane. The motion information prediction network is similar to the fully convolutional structure. It predicts a velocity vector for each BEV grid and combines it with historical frame features to capture the target's motion trend, achieving dynamic target velocity estimation.

[0114] S604. Applicable network structure: The three-dimensional target detection unit adopts a center-point-based three-dimensional target detection network structure. This network structure includes three main modules: a feature extraction module, a feature fusion module, and a detection head module. The feature extraction module extracts top-view features through multi-layer convolution, the feature fusion module performs multi-scale fusion through a feature pyramid network, and the detection head module performs target detection and parameter prediction through a fully convolutional network.

[0115] The 3D object detection unit can employ existing or subsequently developed 3D object detection network structures, including but not limited to VoxelNet (End-to-End Learning for Point Cloud Based 3D Object Detection), CenterPoint (Center-based 3D Object Detection and Tracking), and PV-RCNN (Point-Voxel Feature Set Abstraction for 3D Object Detection). The 3D object detection unit achieves category prediction and 3D bounding box regression through end-to-end training. Furthermore, the network structure can be extended or optimized according to scene requirements to enhance the detection capabilities for dense targets, occluded targets, and high-speed moving targets.

[0116] Figure 3 This is a visualization of the single-frame detection results of the model proposed in this invention, through... Figure 3 You can intuitively see the detection effect of the model proposed in this invention.

[0117] In summary, this invention effectively overcomes the shortcomings of the prior art and has high industrial applicability. The above embodiments are intended to illustrate the substantive content of this invention, but are not intended to limit the scope of protection of this invention. Those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of this invention without departing from the essence and scope of protection of this invention.

[0118] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A BEV perception method that integrates global scene and local instance features, characterized in that, The steps are as follows: S1. Construct a two-layer feature extraction module, including a global scene feature extraction unit and a local instance feature extraction unit. The global scene feature extraction unit extracts a two-dimensional top-down representation covering the entire scene from the multi-view image sequence and retains the spatial structure information of the scene to obtain global scene features; the local instance feature extraction unit obtains fine-grained features of individual targets from the image to obtain local instance features. S2, Feature Fusion: Weighted combination of local instance features and global scene features, and high-density instance-level top-down feature representation is generated through spatial alignment operation; S3. Perform 3D target recognition on the instance-level top-view feature representation, obtain the probability of belonging to different target categories, and output the 3D position information, size, orientation and motion parameters of each target.

2. The BEV perception method integrating global scene and local instance features according to claim 1, characterized in that, The global scene feature extraction unit includes a multi-view image coding unit and a depth estimation unit. The multi-view image coding unit processes the input multi-view image sequence through a shared encoder to extract multi-scale features of each frame to preserve the spatial structure information and texture details of the image. The multi-view image coding unit supports cross-view feature consistency and captures local and global information through convolution or Transformer modules. The depth estimation unit establishes a mapping relationship between the multi-scale features obtained by the multi-view image coding unit and the three-dimensional geometric space to predict the depth probability distribution of each pixel. By combining multi-scale features with depth probability distribution, and using outer product operation or depth weighted mapping to perform dimensionality upscaling operation, pseudo-3D point cloud features are generated. A spatial mapping unit is established, and the pseudo-3D point cloud is mapped to the sensor coordinate system or vehicle coordinate system through camera intrinsic parameters and coordinate transformation. The 3D features are projected onto the BEV plane through top-view feature pooling operation to generate a preliminary top-view feature representation. The preliminary top-view feature representation generated in the previous T-1 frame is spatiotemporally aligned and fused to the preliminary top-view feature representation of the current frame T through a time encoder to generate global top-view features. Global average pooling is performed on the global top-down features to extract the overall semantic representation, resulting in global scene features that include scene structure and dynamic information.

3. The BEV perception method integrating global scene and local instance features according to claim 2, characterized in that, When using convolution, small-sized convolutional kernels slide within the local receptive field to extract detailed features of edges and textures. As the network depth increases, the receptive field expands layer by layer, gradually aggregating a wider range of contextual information to achieve modeling from local to global. When using Transformer, images in a multi-view image sequence are divided into patches and flattened into a sequence. A self-attention mechanism is used to calculate the correlation between any positions, enabling each position to directly interact with all positions globally. This allows for the one-time modeling of long-distance dependencies and the capture of global information. At the same time, multi-head attention and hierarchical structure are combined to take into account both local details and global semantic information. The shared encoder is a shared deep convolutional neural network encoder that extracts features from multi-view images to obtain image feature representations with multi-scale hierarchical structures. The ResNet101 network is used as the shared deep convolutional neural network encoder, which contains 5 residual blocks, each of which contains multiple convolutional layers and skip connections. The depth estimation unit adopts a fully convolutional neural network, which includes an encoder-decoder architecture. The encoder extracts features through multi-layer convolution and pooling operations, and the decoder restores spatial resolution through deconvolution and skip connections. Multi-scale features are mapped to D depth channels through 1×1 convolution, and the depth probability distribution of each pixel is obtained through softmax normalization. The dimensionality upscaling operation employs an outer product operation, where the feature vector f(u,v) of each pixel (u, v) is expanded along the depth direction d to generate pseudo-3D point cloud features. Where ⊗ represents the outer product operation, Let be the probability value of the depth probability distribution of pixel (u, v) at depth direction d; The spatial mapping unit performs coordinate transformation using camera calibration parameters, including intrinsic parameter matrix K and extrinsic parameter matrix; The top-down feature pooling involves placing 3D points into corresponding BEV mesh cells according to their (x, y) coordinates, in the height dimension. The aggregation operation is performed on the top to compress all three-dimensional features within the same BEV grid into a two-dimensional feature vector, forming a BEV feature map, which is the preliminary top-view feature representation. The temporal encoder adopts a Transformer architecture, which includes 8 encoder layers. Each layer includes a multi-head self-attention mechanism, a feedforward network, and residual connections. The temporal feature fusion unit performs spatiotemporal alignment through a motion compensation mechanism, spatially deforms the top-view feature representation of historical frames to align it with the top-view feature representation of the current frame, and generates the final global top-view feature.

4. The BEV perception method fusing global scene and local instance features according to any one of claims 1-3, characterized in that, The local instance feature extraction unit includes a foreground feature filtering unit and an angle-adjusting attention unit. The foreground feature filtering unit selects effective pixel features based on the candidate target region to suppress background interference and enhance the feature expression capability of the target region. The angle-controlled attention unit dynamically adjusts the feature weights based on the spatial angular relationship between different target instances to distinguish targets at different depths along the same light path, thereby improving the ability to identify occluded and dense targets and obtaining local instance features.

5. The BEV perception method integrating global scene and local instance features according to claim 4, characterized in that, The implementation steps of the foreground feature filtering unit are as follows: The original image is input into a convolutional neural network to obtain a two-dimensional feature map. The corresponding pixel is selected on the image plane of the two-dimensional feature map from multiple perspectives using the three-dimensional reference coordinates of each target instance, and image features are sampled from the corresponding pixel position. In the obtained image features, foreground pixels are filtered, and only foreground pixel features located within the candidate target region are retained to generate high-density instance-level top-down features; Background pixels are masked for pixel features that do not belong to the target region to avoid background interference affecting instance features.

6. The BEV perception method integrating global scene and local instance features according to claim 5, characterized in that, The candidate target region is a candidate bounding box predicted by the target detection head; the target bounding box output by the detection network corresponds to an independent target instance. The 3D center or 3D bounding box corners of the target instance are projected onto the image plane of each viewpoint through the camera extrinsic and intrinsic parameter matrices to obtain pixel coordinates. A sampling window is established within a set radius with the pixel coordinates as the center, and features at the corresponding positions are sampled on the image feature map through bilinear interpolation. The filtering process that retains only foreground pixel features located within the candidate target region is achieved through candidate region mask or ROI bounding box restriction. By using the two-dimensional bounding box or foreground segmentation mask output by the detection network, a binary mask is constructed on the feature map, retaining only pixel features located within the ROI range. By determining whether the pixel coordinates fall within the 2D bounding box or instance segmentation mask of any target instance, if they are not within any target area, they are determined to be background areas. The foreground pixel selection is performed through two-dimensional bounding box constraints: for each target instance, a two-dimensional bounding box is obtained based on the projection of the three-dimensional bounding box onto the image plane, only the pixel features located within the two-dimensional bounding box are retained, and pixels that do not belong to the target area are filtered out to generate high-density instance-level top-down features. The background pixel masking is achieved by setting a masking mechanism: for each pixel position, if it is not within the bounding box of any target instance, its feature value is multiplied by a weight decay factor of 0.

1.

7. The BEV perception method fusing global scene and local instance features according to claim 5 or 6, characterized in that, The implementation steps of the angle-adjustable attention unit are as follows: The angular deviation on the BEV plane is calculated based on the center coordinates in the three-dimensional space of each pair of target instances; In the self-attention mechanism, the attention weights between each target instance are dynamically adjusted using angle deviation; The attention weights after angle adjustment are applied to the instance-level top-down features, which enable the instance-level top-down features to have a stronger ability to identify target instances at different depths, generating depth-discriminating features, which are local instance features.

8. The BEV perception method integrating global scene and local instance features according to claim 7, characterized in that, The method for dynamically adjusting the attention weight is as follows: for instances belonging to the same target instance or spatially adjacent instances, the attention connection between feature vectors is enhanced, that is, the similarity weight in self-attention calculation is used to make the related instance-level top-down features mutually reinforce information transmission; for instances located at different depths or on different light paths, the attention weight is reduced to suppress feature interference. A multi-head attention structure is adopted, which sets up multiple parallel attention heads to model the correlation between target instances in different feature subspaces, and adjusts the attention weights by combining the angle difference between target instances; The target instance predicted by the 3D target detection network or BEV detection head includes 3D center coordinates, length, width, height, and orientation information; for instance i and instance j, the 3D center coordinates are respectively... and Calculate the azimuth angle relative to the sensor origin: , Angular deviation is The value range is [-π, π]. It is the arctangent function in the four quadrants; The attention weight adjustment is achieved through an angle control function. The implementation is as follows: where σ is the angular bandwidth parameter, and the final attention weights are calculated as follows: ,in This is a correlation function based on feature similarity. This is used to normalize the attention weights, ensuring that the sum of the weights for the same instance query and all keys is 1. For the query characteristics of instance i, Here are the key features of instance i, used for similarity matching with the query; the final output features of instance i are: ;in, Represents the value characteristics of the target instance; The depth differentiation feature is achieved by enhancing the feature differences between targets at different depths along the same angular direction. It classifies the angle of the instance center on the BEV plane, identifying multiple target instances along the same angular direction, and then combines this with depth... Coordinates are used for depth differentiation; for multiple targets in the same angle direction, a depth-weighted mechanism is used to distinguish them: ； in, For depth weight parameters, and These are the mean and standard deviation of the depth, respectively. The three-dimensional center depth of the target instance. This is an instance-level top-down feature after angle adjustment. This ultimately enhances the instance-level top-down features, which provide greater depth discrimination capabilities.

9. The BEV perception method integrating global scene and local instance features according to claim 8, characterized in that, The feature fusion steps are as follows: For each target instance, extract instance-level top-down features at different resolutions to generate multi-scale instance features with high and low resolutions; Spatiotemporally align multi-scale instance features with instance-level top-down features of historical frames from the previous T-1 frames; An adaptive feature fusion strategy is used to perform weighted fusion of multi-scale instance features and instance-level top-down features of historical frames.

10. The BEV perception method fusing global scene and local instance features according to claim 9, characterized in that, The multi-scale instance feature extraction is achieved through a feature pyramid network, which includes four scale levels. The spatiotemporal alignment uses motion compensation or optical flow estimation to correct the displacement of the target in consecutive frames, and maps the instance-level top-down features of historical frames to the reference coordinate system of the current frame through coordinate transformation. The adaptive feature fusion strategy adopts an adaptive fusion method based on attention or weighting strategy, which weights and combines the multi-scale instance features of the current frame with the instance-level top-down features aligned with historical frames to generate a high-density, information-rich instance-level top-down feature table. The adaptive feature fusion strategy is implemented through a learnable fusion weight network. The fusion weight network adopts a fully convolutional neural network structure. The input is the concatenation of multi-scale instance features and historical top-down features of historical frames, and the output is the fusion weight. The 3D target recognition predicts the target category probability for each grid or anchor point of the input instance-level top-down features using a fully convolutional neural network, Transformer structure, or multi-head attention mechanism; and obtains the confidence score of each category through softmax to quantify the probability that the grid / anchor point belongs to different target categories. A fully convolutional neural network contains multiple convolutional layers and the ReLU activation function; The bounding box prediction network outputs 3D bounding box parameters for each target, including the target center position, target size, orientation angle, and velocity vector. By utilizing instance-level top-down features aligned with historical frames and a motion information encoding module, the dynamic behavior of the target is captured, ensuring accurate 3D positioning and motion description of both static and dynamic targets. Historical frame features are input through the BEV features of the previous T-1 frames obtained by the temporal feature fusion unit in step S204. The motion information encoding module extracts dynamic information such as the target's displacement and velocity trends, which is then fused with the top-down features of the current frame to model the target's dynamic behavior. The bounding box prediction network is based on a fully convolutional network and predicts the target center, target size, and target orientation angle in the BEV plane for each grid or anchor point. By using a motion information prediction network to predict the velocity vector for each grid and combining it with instance-level top-down features from historical frames to capture the target's motion trend, dynamic target velocity estimation is achieved.

Images