A multi-camera three-dimensional perception method based on image pre-fusion

Abstract

The application relates to a multi-camera three-dimensional perception method based on image pre-fusion, comprising the following steps: based on multi-sensor registration, acquiring the internal and external parameters of each view angle of a multi-camera system, and adopting a linear grid method to positionally encode a view cone space; through a double-channel network and multi-view image feature fusion, obtaining multi-view features sensitive to three-dimensional positions; adopting two-dimensional key point detection and feature sampling to acquire foreground target features, explicitly assisting in depth detection through additional depth prediction, and acquiring implicit depth estimation through multi-dimensional depth aggregation; inputting the multi-view deep features into a decoder to obtain multi-view three-dimensional image perception results. Compared with the prior art, the method of explicit depth estimation and implicit depth estimation can effectively avoid depth ambiguity, that is, the phenomenon that three-dimensional depth falls into the same pixel area, leading to inaccurate depth estimation, while the multi-view three-dimensional perception performance of a vehicle end is improved, and the design is intuitive and effective.

Description

Technical Field

[0001] This invention relates to the field of perception technology for autonomous vehicles, and in particular to a multi-camera 3D perception method based on image front fusion. Background Technology

[0002] With the continuous development of autonomous driving technology, intelligent driving has gradually integrated into daily traffic environments. Complex urban roads are one of the important application scenarios for intelligent driving. For autonomous driving perception systems, accurately perceiving traffic participants on urban roads is a crucial and challenging task, and it is also the foundation for autonomous vehicles to make safe and effective driving decisions. Multi-view images enable visual perception systems to understand and perceive three-dimensional scenes more comprehensively and accurately. However, multi-view images introduce the problem of multi-sensor fusion, becoming a challenge for practical application.

[0003] In terms of perception, there has been much debate regarding the selection and fusion of different sensors. Mainstream commercial solutions are primarily divided into pure vision multi-camera solutions and point cloud fusion solutions combining vision and LiDAR. Due to the high cost and deployment difficulty of LiDAR sensors, many researchers in the autonomous driving field rely solely on visual information for 3D road perception. 3D target perception based on pure vision camera data, compared to point cloud data, offers advantages such as lower cost, wider detection range, and richer semantic information, making it an increasingly popular research area. Multi-view detection requires feature fusion using multiple sensors to achieve superior results compared to single-view vision detection. Multi-camera systems typically require specific settings to ensure overlapping sensor fields of view. Currently, an increasing number of autonomous vehicles are equipped with multiple onboard cameras for road environment perception systems. Multi-camera systems can enable lightweight, low-cost, and more easily mass-producible autonomous driving systems, reducing the cost of using autonomous driving and facilitating its rapid deployment and maturity. However, in practical applications, depth ambiguity issues in image detection—where 3D depth falls into the same pixel region, leading to inaccurate depth estimation—result in low accuracy in multi-view 3D detection, failing to effectively improve the performance of onboard multi-view 3D perception. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art by providing a multi-camera 3D perception method based on image pre-fusion, which can effectively avoid depth ambiguity and improve the accuracy of multi-view 3D detection.

[0005] The objective of this invention can be achieved through the following technical solution: a multi-camera 3D perception method based on image pre-fusion, comprising the following steps:

[0006] S1. Based on multi-sensor registration, acquire the intrinsic and extrinsic parameters of each viewpoint of the multi-camera system, and use the linear grid method to encode the position of the view frustum space.

[0007] S2. By fusing dual-channel network with multi-view image features, three-dimensional position-sensitive multi-view features are obtained;

[0008] S3. Use two-dimensional keypoint detection and feature sampling to obtain foreground target features, explicitly assist depth detection through additional depth prediction, and obtain implicit depth estimation through multi-dimensional depth aggregation;

[0009] S4. Input the multi-view deep features into the decoder to obtain the multi-view 3D image perception results.

[0010] Furthermore, step S1 specifically includes the following steps:

[0011] S11. Using a linear 3D mesh generation algorithm, the view frustum space is discretized into multiple meshes to obtain the coordinates in the view frustum meshes from different viewpoints.

[0012]

[0013] Where K is the camera's intrinsic parameter matrix, specifically a 3×3 matrix representation;

[0014] S12. Regularize the points in the three-dimensional space to ensure that the positional grid representation of the frustum space from different viewpoints is the same:

[0015]

[0016] Among them, [x min ,y min ,z min ] and [x max ,y max ,z max [ ] represents the detectable space region with the vehicle as the origin. Targets outside this region will be filtered out. The regularized 3D position points are represented in tensor form as follows:

[0017] Furthermore, step S2 specifically involves representing three-dimensional spatial location points as high-dimensional location grid feature codes through location feature embedding:

[0018]

[0019] Wherein, ψ(·) is the location encoding module calculation function related to the two-dimensional image.

[0020] Furthermore, step S3 specifically includes the following steps:

[0021] S31. Train a subnetwork including two-dimensional depth, two-dimensional key points, and two-dimensional size through an auxiliary two-dimensional supervision network for assisted perception.

[0022] S32. Sampling foreground target features based on two-dimensional center point;

[0023] S33. Design a multi-dimensional depth aggregation module to predict different depth distributions as depth confidences for depth-ambiguous location features. For each image feature, use the depth confidence as the depth weight to redistribute the feature to obtain multi-scale features with depth representation.

[0024] Furthermore, the training loss of the auxiliary supervision network in step S31 includes depth loss, two-dimensional keypoint loss, and geometric loss, wherein the depth loss is specifically:

[0025]

[0026] The specific loss of two-dimensional key points is as follows:

[0027] L G =-|yQ| β ((1-y)log(1-Q)+ylog(Q))

[0028] Geometric loss is specifically L1 loss.

[0029] Furthermore, the sampling formula in step S32 is:

[0030]

[0031] Where γ is the balanced sampling coefficient, specifically, the feature positions of the first ρ confidence levels on the feature map are selected as sampling points.

[0032] Furthermore, the specific process of step S33 is as follows:

[0033] S331. Multiple MLP networks and forward paths are used for depth representation, and linear layer depth representation is used for dimensionality reduction to obtain sampling weights.

[0034] S332. Use bilinear interpolation to obtain weighted sampling points. Through implicit depth representation, the depth on the pixel frustum is discretely sampled. The depth information at the position with higher confidence is taken as the basis for 3D depth point prediction. This prevents the ambiguity of the network's depth sampling at a large range of positions from causing large deviations in the 3D information estimation.

[0035] Furthermore, the sampling weight in step S331 is:

[0036]

[0037] p seudu =ψ loc3d (Q l ), l=1,2,…,L

[0038] Where, p I For a predefined depth grid, D I The number of depth grids is represented by , and Norm represents regularization. Further, the implicit depth sampling formula in step S332 is:

[0039] C m =Bilinear(ψ) loc3d (W dep ),p seudo )

[0040] The formula for deep feature aggregation is:

[0041] Q agg =C m ·Q l .

[0042] Furthermore, the decoder decoding formula in step S4 is as follows:

[0043] Q l =Ω l (F 3d Q l-1 ), l=1,…,L

[0044] Among them, Q l It is the decoding layer with index l in the decoder, Q l ∈R M×C It is the Query tensor output after being updated by the Lth layer of the decoder network. M and C represent the number of defined Queries and the number of channels, respectively. In each decoding layer, the Query and the image features that are fused with 3D position coordinate information are cross-attentioned through a multi-head attention network. After multiple layers of networks update the content features and position features in the Query, the updated Query will obtain high-dimensional features, which are used for subsequent classification and regression head networks to finally obtain the target in the 3D scene.

[0045] Compared with the prior art, the present invention has the following advantages:

[0046] This invention first utilizes a multi-view approach, employing the camera intrinsic and extrinsic parameter matrices obtained from the multi-camera registration process to encode the position of the 3D view frustum region of the multi-camera system. This positional encoding is then used as 3D positional information and merged with image elements in the subsequent image fusion step. Secondly, the position-sensitive image features obtained after fusion are used as input to the multi-view perception decoder. Through feature sampling and a multi-dimensional depth aggregation module in the decoder, explicit and implicit depth-sensitive multi-view image features are acquired. Finally, a deep neural network decodes the 3D dynamic target information surrounding the autonomous vehicle. Therefore, for multi-camera perception in autonomous vehicles, the use of explicit and implicit depth estimation methods effectively avoids depth ambiguity, preventing inaccurate depth estimation due to 3D depths falling into the same pixel region, while simultaneously improving the multi-view 3D perception performance of the vehicle.

[0047] This invention proposes a 3D target detector based on multi-view location encoding. This detector employs a pre-image fusion approach, enabling accurate 3D target detection across multiple viewpoints. The 3D location encoding process samples the 3D points of the view frustum in each viewpoint, fusing the 3D location information with 2D image features, and then extracts multi-view image features using a Transformer network. This detector can directly output 3D bounding boxes in the BEV (Bird's Eye View) without requiring additional post-processing.

[0048] This invention proposes a multi-dimensional depth aggregation module, employing a similar approach to explicit depth estimation. In the feature decoder module, a pseudo-depth representation is estimated to resolve the depth ambiguity problem in converting 2D feature maps to 3D depth. Using pseudo-depth for feature aggregation from the input feature map effectively helps the network acquire more information about 3D estimation. Furthermore, a supervised 3D object detector is proposed. Utilizing a supervised module, the estimation of 2D scale information enhances the network's ability to detect 3D foregrounds. Simultaneously, an additional depth supervision network is trained using ground truth depth data from laser point clouds, further assisting the network in perceiving 3D depth information. Attached Figure Description

[0049] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0050] Figure 2 This is a schematic diagram of the multi-view fusion network structure based on position encoding in this invention;

[0051] Figure 3 This is a schematic diagram of the multidimensional deep aggregation module process in this invention;

[0052] Figure 4 This is a schematic diagram of the multidimensional deep aggregation module structure in this invention;

[0053] Figure 5 This is a schematic diagram of the overall structure of the invention, including the addition of an auxiliary supervision module and sampling.

[0054] Figure 6 This is a schematic diagram of the structure of the location encoding and multi-view image feature fusion network in this invention;

[0055] Figure 7 This is a visualization of the explicit depth supervision in the example.

[0056] Figure 8 This is the perception result for cross-view targets in the example;

[0057] Figure 9 This is the perception result for non-cross-view targets in the example. Detailed Implementation

[0058] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0059] Example

[0060] like Figure 1 As shown, a multi-camera 3D perception method based on image pre-fusion includes the following steps:

[0061] S1. Based on multi-sensor registration, acquire the intrinsic and extrinsic parameters of each viewpoint of the multi-camera system, and use the linear grid method to encode the position of the view frustum space.

[0062] S2. By fusing dual-channel network with multi-view image features, three-dimensional position-sensitive multi-view features are obtained;

[0063] S3. Two-dimensional keypoint detection and feature sampling are used to obtain foreground target features. Additional depth prediction is used to explicitly assist depth detection, and multi-dimensional depth aggregation is used to obtain implicit depth estimation, thereby improving the performance of 3D perception.

[0064] S4. Input the multi-view deep features into the decoder to obtain the multi-view 3D image perception results.

[0065] Specifically, such as Figure 2 As shown, in step S1, the multi-view original image data is input into the feature encoder to obtain downsampled multi-view image features, such as... Figure 2 The process within the left-hand box shows that the downsampling operation reduces the size of the feature map while decreasing the amount of data input to the decoder, thereby reducing the overall number of network parameters. Figure 2 The lower middle box illustrates the 3D position encoding process. Based on multi-sensor registration, it acquires the intrinsic and extrinsic parameters of each viewpoint from the multi-camera system and uses a linear mesh method to encode the position in the view frustum space. The camera view frustum space is then meshed to obtain (W...).F H F Encoding in 3D dimension. Each element represents Where (u j ,v j ) represents the pixel coordinates in the pixel coordinate system, d j This is the depth value along the depth axis, which is orthogonal to the image coordinate system. Coordinates in the frustum grid at different viewpoints. Projected into 3D space

[0066]

[0067] Where K is the camera's intrinsic parameter matrix, which is represented as a 3×3 matrix.

[0068] To ensure that the view frustum spaces from different perspectives have the same positional mesh representation, the points in the 3D space need to be regularized:

[0069]

[0070] Where [x] min y min , z min ] and [x max y max , z max [ ] represents a region of detectable space with the vehicle as the origin; targets outside this region will be filtered. The regularized 3D location points are represented in tensor form as follows:

[0071] In step S2, the 3D view frustum mesh is positionally encoded as supplementary positional information in the Transformer network. This encoding uses positional feature embedding.

[0072] Q l =Ω l (F 3d Q l-1 ), l=1,…,L

[0073] The formula for calculating the high-dimensional location grid feature encoding that represents a three-dimensional spatial location point through location feature embedding is as follows:

[0074]

[0075] Wherein, ψ(·) is the two-dimensional image-related position encoding module designed in this invention, and its structure is as follows: Figure 6As shown, unlike 3D location encoding methods that separate image features, this module implicitly assists the 3D spatial location feature network by utilizing the 2D features of multi-view images. For example, the depth of an object is related to its size in the image. This information makes the network more attentive to instance-related information, providing some prior knowledge for learning content feature queries. Simultaneously, this module also helps to align the 3D location with the image pixel features as closely as possible in terms of feature dimensions. At this point, the construction of multi-view image feature-related location encoding is complete.

[0076] In step S3, during the training phase, considering that two-dimensional detection can effectively help improve the performance of three-dimensional perception, this invention uses an auxiliary supervision network to supervise two-dimensional attributes during training, such as... Figure 5 As shown, the training loss of the auxiliary supervised network includes:

[0077] Deep loss —

[0078] Two-dimensional keypoint loss - L G =-|yQ| β ((1-y)log(1-Q)+ylog(Q));

[0079] The geometric loss is L1 loss.

[0080] Deep supervision networks, center points, and 2D bounding box networks are used for auxiliary supervision. At the same time, the results of category supervision and center points are used for inference to sample foreground target features on multi-view feature maps, so that the network pays attention to the foreground target.

[0081] During the network inference phase, the feature map is sampled based on the category and centroid network output. The sampled feature map is denoted as F. s :

[0082]

[0083] Where γ is the balanced sampling coefficient. This invention selects the feature locations with the first ρ confidence levels as sampling points on the feature map.

[0084] In step S3, a multi-dimensional depth aggregation module is also used on the multi-view image features to extract implicit depth features and eliminate the influence of depth ambiguity. The specific working principle of this module is as follows: Figure 3 As shown, the method of this invention predicts different depth distributions as depth confidences for depth-ambiguous location features. For each image feature, this depth confidence is used as the depth weight for feature reallocation to obtain multi-scale features with depth representations. This module performs pseudo-depth extraction on image features at a self-attention layer, regresses the latent depth representation of each anchor, and then implicitly samples image features based on the depth representation.

[0085] This module contains multiple MLP networks and a feedforward path for depth representation, and uses linear layer depth representation for dimensionality reduction to obtain sampling weights:

[0086]

[0087] p seudu =ψ loc3d (Q l ), l=1,2,…,L

[0088] Where, p I For a predefined depth grid, D I This indicates the number of depth grids, and Norm represents regularization.

[0089] This weight is ultimately multiplied by the input content features to achieve an effect similar to deformable attention. The implicit depth sampling formula is:

[0090] C m =Bilinear(ψ) loc3d (W dep ),p seud o)

[0091] The formula for deep feature aggregation is:

[0092] Q agg =C m ·Q l

[0093] The structure of the deep aggregation module is as follows Figure 4 As shown. This implicit depth representation method can discretely sample the depth on the pixel frustum and take the depth information at the position with higher confidence as the basis for 3D depth point prediction, preventing the ambiguity of the network's depth sampling at a large range of positions from causing large deviations in the 3D information estimation.

[0094] For depths of non-sampling points, the fusion weight approaches zero.

[0095] Finally, in step S4, the multi-view image features obtained by the multi-layer pre-fusion method are fed into the decoder to calculate the perceived 3D dynamic foreground target. The decoder decoding formula is as follows:

[0096] Q l =Ω l (F 3d Q l-1 ), l=1,…,L

[0097] Q l It is the decoding layer with index l in the decoder, Q l ∈R M×CThis is the Query tensor output after being updated by the l-th layer of the decoder network, where M and C represent the number of defined Queries and the number of channels, respectively. In each decoding layer, the Query and the image features fused with 3D position coordinate information undergo cross-attention operation through a multi-head attention network. After multiple layers of networks update the content features and position features in the Query, the updated Query will obtain high-dimensional features for subsequent classification and regression head networks to obtain the target in the 3D scene.

[0098] c li =Φ ref (q li )

[0099] Where Φ ref It is a neural network detection head, c li This represents the classification result of the i-th element of the Query output from layer l. Through a multi-head attention network, the model can learn the relationships between 2D features from multiple perspectives. Each sampling layer focuses on the Query tensor Q. l The content is embedded and updated. This updated information interaction can be expressed by the formula:

[0100] Q = F' B *M q ,K=F' B *M k V=F' B *M v ,

[0101]

[0102] Where F′ B It is F B The resulting one-dimensional sequence after stretching has dimension c. B In the formula, Q, K, and V are obtained by performing linear transformations on the input 2D features, representing the query, key, and value, respectively. The features output by a single layer of feature encoding can then be represented as:

[0103] F out =Φ MLP (A)+F' B

[0104] By stacking multiple self-attention layers, deeper content encoding can be obtained, improving the network's ability to extract features.

[0105] This embodiment uses multi-view vehicle image data from the nuScenes dataset to conduct a case study analysis on dynamic target perception in the road environment.

[0106] First, during the training phase, the input images are preprocessed, and target data within the range of [-51.2, -51.2, -5.0] to [51.2, 51.2, 3.0] in the bird's-eye view scene from the vehicle are selected for training and validation.

[0107] Targets outside the selection range are filtered out, retaining only those within the range. Simultaneously, targets projected onto each viewpoint image are preprocessed to extract useful information, reduce noise, and provide better input for subsequent model training, including:

[0108] 1. Random cropping and scaling are used to adjust image size, enhancing the network's generalization performance. Irrelevant regions can also be cropped. This reduces computational complexity and improves the efficiency of model training and inference.

[0109] 2. Randomly apply geometric transformations to the images, including translation, rotation, and mirroring, to increase the diversity of the training data and improve the model's robustness. Expand the dataset to better adapt the model to various scenarios. Rotate the image randomly within the range. Scale randomly within the range [0.95, 1.05].

[0110] 3. Image normalization: The images are normalized using the mean [123.675, 116.28, 103.53] and variance [58.395, 57.12, 57.375] to enable the model to converge quickly. This parameter is inferred from the average pixel values ​​of the images in the dataset.

[0111] 4. Image padding is used to meet the network's requirements for image size after geometric transformations such as rotation and translation. In this embodiment, the padding value is uniformly set to zero.

[0112] 5. Preprocessing of various labeled true values, construction of camera intrinsic and extrinsic parameter matrices, transformation of rotational angular coordinate system, transformation of depth coordinate system, etc.

[0113] In network inference output, the output visualization of explicit deep supervised networks is as follows: Figure 7 As shown, the image is divided into rows representing images from different forward-facing perspectives, and columns representing the original image, ground truth depth image, and a visualization of the high-dimensional tensor output by the depth network. It can be seen that the depth output distribution in the third row is close to the ground truth depth image in the second row. The portion indicated by the red box is basically similar to the ground truth distribution, with the positional offset representing the resolution error after downsampling. It can be inferred that the depth prediction network, by estimating the depth of the monocular image, can assist the 2D feature extractor in obtaining features related to 3D depth to a certain extent, and the feature extractor part has been optimized for 3D object detection.

[0114] Figure 8 and Figure 9The visualization results of this embodiment are shown, where orange bounding boxes represent vehicles, blue bounding boxes represent pedestrians, and red bounding boxes represent bicycles. It can be observed that targets in most urban road traffic scenes can be successfully and accurately detected. In the bird's-eye view in the fourth row, the purple line on the left represents the predicted target velocity, and the purple line on the right represents the ground truth range of the point cloud. It can be observed that the detector in this chapter can also perform well in detecting targets that are relatively far from vehicles. Figure 8 This demonstrates that when the same target appears in multiple viewpoints, the proposed multi-view pre-fusion detection algorithm can effectively detect such targets without requiring additional post-processing. When viewed from a bird's-eye view, targets existing in images from multiple viewpoints can be accurately identified as the same target, which is the advantage of pre-fusion algorithms over post-fusion algorithms.

[0115] In summary, this scheme proposes a depth feature aggregation module based on a multi-view location encoding 3D object detector to mitigate the impact of depth ambiguity on 3D detection. This module samples multi-view content and location features by predicting pseudo-depth, and then aggregates the sampled features with the original features in a multi-dimensional manner, enabling the network to learn features that are more sensitive to 3D perception. This scheme also enhances the model's 3D perception capability through auxiliary supervision and sampling networks, resulting in a significant improvement in the vehicle-side multi-view 3D perception performance.

Claims

1. A multi-camera 3D perception method based on image pre-fusion, characterized in that, Includes the following steps: S1. Based on multi-sensor registration, acquire the intrinsic and extrinsic parameters of each viewpoint of the multi-camera system, and use the linear grid method to encode the position of the view frustum space. S2. By fusing dual-channel network with multi-view image features, three-dimensional position-sensitive multi-view features are obtained; S3. Use two-dimensional keypoint detection and feature sampling to obtain foreground target features, explicitly assist depth detection through additional depth prediction, and obtain implicit depth estimation through multi-dimensional depth aggregation; S4. Input the multi-view deep features into the decoder to obtain the multi-view 3D image perception results; S3 specifically includes the following steps: S31. Train a subnetwork including two-dimensional depth, two-dimensional key points, and two-dimensional size through an auxiliary two-dimensional supervision network for assisted perception. S32. Sampling foreground target features based on two-dimensional center point; S33. Design a multi-dimensional depth aggregation module to predict different depth distributions as depth confidences for depth-ambiguous location features. For each image feature, use the depth confidence as the depth weight to redistribute the feature to obtain multi-scale features with depth representation. The specific process of S33 is as follows: S331. Multiple MLP networks and forward paths are used for depth representation, and linear layer depth representation is used for dimensionality reduction to obtain sampling weights. S332. Use bilinear interpolation to obtain weighted sampling points. Through implicit depth representation, discretely sample the depth on the pixel frustum and take the depth information at the confidence high position as the basis for 3D depth point prediction. For depths of non-sampling points, the fusion weight approaches zero.

Images