Vehicle exterior multi-view fusion target detection method and system

Abstract

The invention discloses a method for detecting a multi-view fusion target outside a vehicle. The method comprises the following steps of: 1, acquiring a multi-view image data set of an automatic driving vehicle, and preprocessing an image to obtain a multi-view image data set; 2, performing fusion scale feature extraction on the multi-view image; 3, constructing a multi-view fusion target detection model based on a bilinear sampling method based on features extracted from fusion scale features; 4, training the multi-view fusion target detection model by using the multi-view image data set; and 5, performing multi-scale target detection by using the trained multi-view fusion target detection model. The method is few in training parameters, low in training cost and high in detection precision, and has a good detection effect on a multi-scale target.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of automatic driving visual perception technology and multi-target detection, and particularly relates to a multi-view fusion target detection method and system outside a vehicle. BACKGROUND

[0002] The perception technology based on a visual sensor is called the eye of an automatic driving vehicle, and the accuracy of perception in an outdoor scene is crucial and even directly threatens the personal safety of a driver, especially in the case of pedestrian and vehicle target detection. Initially, the detection direction for pedestrians and vehicles is limited to target detection in a single view, and the actual detection effect is not satisfactory. In recent years, target detection based on a bird's eye view (BEV) has been very popular in the field of automatic driving, which fuses information of various views into a space to form a bird's eye view, greatly improving the precision and robustness of a target detection model.

[0003] At present, mainstream target detection algorithms are based on the BEV view, but there are many construction methods for the BEV view, and these different construction methods directly affect the effect of the BEV target detection model. The BEV space can fuse various feature information, such as image information of a camera, point cloud information of a laser radar and feature information of a radar. However, due to the high economic cost of a laser radar and the too sparse presentation of radar data, a pure visual BEV solution is increasingly favored by the industry and the scientific research community. In the early stage, a pure visual construction method uses a depth estimation network to discretize a depth interval in a classification manner to predict a depth value of each pixel point, obtains a depth distribution probability of each pixel, and projects perspective view features onto the BEV view through voxel pooling according to the depth distribution of each pixel. Later, due to the proposal of a Transformer model, researchers have applied the Transformer to a target detection model in a pure visual view, and greatly improved the detection precision by using spatial attention and temporal attention.

[0004] However, neither the method based on depth estimation nor the method based on depth learning considers the scale characteristics of pedestrians, vehicles and bicycles in a real scene, and cannot take into account the task of multi-scale target detection.

[0005] The FCN used in a traditional multi-scale detection task generates a fixed scale and cannot cover a larger scale range, and also cannot generate an intermediate layer. Later, methods such as SFPN improve the shortcomings to generate an intermediate layer, but also do not consider the application of spatial attention and are difficult to generate more rich feature information. In addition, the model of a depth learning method often has a large amount of training parameters, consumes a large amount of computing resources in actual operation, and cannot adapt to the size of the computing power of an existing automatic driving vehicle.

[0006] Prior art 1:

[0007] US10,839,234 B2 (System and method for three-dimensional object detection), which discloses obtaining a two-dimensional image through a camera and estimating the length, width, height, position and orientation information of an object in three-dimensional space (especially in the vehicle coordinate system) with the help of the camera's calibration matrix.

[0008] Technical problems:

[0009] This method mainly relies on single camera image and its calibration data to realize three-dimensional space positioning, and lacks consideration of multi-view fusion, and does not utilize the geometric consistency between multiple cameras for end-to-end alignment. This approach may result in insufficient fusion accuracy and boundary consistency in application scenarios that require multi-view fusion and high spatial consistency, which is not conducive to high-precision multi-view three-dimensional object detection.

[0010] Approaching prior art 2:

[0011] US11,221,413 B2 (Three-dimensional object detection system fusing image and LiDAR features using continuous convolution), which innovatively fuses camera image and LiDAR point cloud features, and uses high-precision map or geometric semantic map data as an aid to fuse multi-modal information in BEV (bird's eye view) space.

[0012] Technical problems:

[0013] Although this method realizes deep fusion of multi-modal sensor information (LiDAR and camera), its core still relies on LiDAR data to generate BEV features. Its applicability is limited in scenarios that rely on optical images and lack high-quality LiDAR. In addition, this method does not perform optimization for fine-grained geometric alignment between multiple camera images, making it difficult to meet the demand for precise fusion of all off-board views in a unified BEV space.

[0014] These two patents reflect the common path and focus of the industry in three-dimensional object detection, but there is still a significant gap compared to the technical solution you proposed in terms of handling multi-view geometric consistency, LiDAR dependency, and fusion end-to-end alignment. SUMMARY

[0015] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present application is to provide a multi-view fusion object detection method and system for off-board, which has fewer training parameters, low training cost, and high detection accuracy, and has good detection effect on multi-scale targets.

[0016] In the environmental perception of autonomous driving systems, multi-view fusion-based three-dimensional target detection has always faced the bottleneck of engineering deployment. The fundamental problem is that the registration error caused by the time asynchronization between multi-camera views, extrinsic parameter drift and optical distortion in dynamic scenes. Existing methods mostly use offline calibration and fixed extrinsic parameter matrix, which cannot adapt to the micro pose drift caused by vehicle body vibration and temperature difference, resulting in spatial misplacement when the vehicle turns or bounces. This error will be amplified to tens of centimeters of three-dimensional positioning deviation under the condition of millimeter-level extrinsic parameter deviation, directly affecting the detection accuracy and multi-frame temporal consistency in the BEV space. Therefore, establishing a multi-view fusion detection mechanism with self-calibration and time delay compensation capability is the key path to solve the stability and reliability of multi-camera collaborative perception.

[0017] In the dynamic calibration process of the multi-camera system, an extrinsic parameter adaptive updating method based on sliding window optimization is used, and structural environmental features such as lane lines and ground planes are introduced into the constraints. Through the joint optimization of short-time pose prediction provided by the IMU and the visual homography matrix, the spatial transformation matrix between cameras can be corrected in real time. This process solves the perturbation in the Lie algebra domain, and combines sparse Levenberg-Marquardt iteration to realize the continuous and smooth updating of the extrinsic parameter increment. This mechanism effectively reduces the long-term accumulated error caused by temperature drift and assembly tolerance, so that the multi-view geometric relationship remains consistent in different driving stages, providing high-precision geometric priors for subsequent feature fusion.

[0018] In the multi-view feature fusion layer, traditional bilinear interpolation and voxelization sampling have the problems of fixed sampling kernel, resolution loss and geometric distortion. The adaptive four-point sampling method proposed in this study learns the distribution of sampling points in the pixel sub-domain through offset prediction, and introduces a learnable elliptical kernel function to realize the adaptive adjustment of optical flow direction and perspective distortion. This method performs difference calculation in the spatial continuous domain, which can significantly improve the sampling accuracy while maintaining computational efficiency. The time domain adaptation module parallel to this maintains the spatio-temporal consistency of the feature space by compensating for the feature drift caused by rolling shutter distortion and vehicle motion in the feature extraction stage.

[0019] In terms of occlusion modeling and feature confidence evaluation, the differentiable Z-buffer mechanism is introduced to calculate the visibility weight between multiple views. By obtaining the occlusion probability map through depth sorting, and combining the confidence map output by the uncertainty estimation head, the cross-view fusion weight is dynamically gated. This mechanism structurally avoids the superposition of false responses caused by low-texture areas and reflective surfaces, enabling the network to maintain stable perception performance in varying lighting conditions, rain and snow reflection, and complex occlusion environments. At the same time, the epipolar consistency constraint further ensures the corresponding relationship of cross-view features in the geometric sense, reducing misalignment and drift.

[0020] In terms of spatial representation structure, a polar coordinate-Cartesian hybrid grid is used to construct the BEV feature plane. The near-field region utilizes a polar coordinate annular structure to enhance angular resolution, while the far-field region employs an equidistant Cartesian grid to ensure spatial uniformity. The two coordinate systems achieve energy conservation in the transition zone through a continuously differentiable resampling function, resulting in a seamless connection of feature distributions. This design ensures full field-of-view coverage for the detection algorithm while balancing high-precision near-field perception with global far-field perception, making it suitable for multi-camera systems and heterogeneous vision modules in autonomous vehicles.

[0021] Finally, the overall system integrating the above modules is trained end-to-end using a joint loss function. The detection subtasks, including heatmaps, offset, size, orientation, and height, work together with epipolar consistency, uncertainty entropy regularization, and extrinsic parameter sparsity constraints to form an optimization framework with both physical and semantic constraints. Real-world testing in multiple scenarios demonstrates high stability under conditions of complex occlusion, uneven lighting, and varying vehicle speeds, with a significant improvement in average detection accuracy. This method can be directly embedded into Level 3 and above autonomous driving perception stacks in industrial applications, providing real-time and reliable extrinsic parameter adaptation and spatial consistency guarantees for the vision subsystem in multi-sensor fusion.

[0022] To achieve the above objectives, the technical solution adopted by the present invention is: a three-dimensional target detection method based on multi-view fusion of vehicle exterior, comprising the following steps:

[0023] S0, based on camera timestamps and vehicle speedometer / IMU information, performs cross-camera time delay estimation and compensation to obtain time-aligned multi-view image sequences;

[0024] S1, acquire the multi-view image dataset of the autonomous vehicle and perform preprocessing;

[0025] S2, based on the online extrinsic self-calibration module, incrementally update the extrinsic parameters of each view. The self-calibration is constrained by lane lines, road surface and short-term visual odometer, and optimizes the SE(3) disturbance within the sliding window.

[0026] S3 utilizes a backbone network with spatial attention mechanism to extract multi-scale features from each view image and compensates for feature distortion caused by rolling shutter and vehicle motion through a temporal adapter;

[0027] S4 employs adaptive four-point sampling (AFQS) to continuously sample learnable anisotropic kernels within the pixel subdomain, and combines visibility estimation and uncertainty weights to complete cross-view fusion, generating a polar coordinate-Cartesian hybrid BEV feature raster.

[0028] S5, input the BEV features into the CenterPoint detection head with height prior and heading continuity regularization, and output the center, size, height and orientation of the three-dimensional target;

[0029] S6 performs end-to-end training on the network based on a joint loss consisting of heatmap loss, offset loss, size loss, orientation loss, height loss, epipolar consistency loss, uncertainty entropy regularization, and extrinsic parameter sparsity regularization.

[0030] Furthermore, the extrinsic parameter self-calibration of S2 is achieved by performing homography constraint fitting on the common lane line features between multiple view frames, combining the short-time pose information obtained by the IMU, using sparse Levenberg-Marquardt optimization to obtain the extrinsic parameter increment Δξ, and updating the initial extrinsic parameter matrix in the form of Lie algebra.

[0031] Furthermore, the adaptive four-point sampling (AFQS) backprojects the pixel coordinates (u,v) for each BEV cell, outputs four sub-displacements {Δu_i, Δv_i} by the offset prediction head, and performs weighted aggregation with a learnable elliptical anisotropic kernel Kθ, thereby achieving high-precision feature fusion in continuous space.

[0032] Furthermore, the visibility estimation generates an occlusion probability map M through a differentiable Z-buffer, which together with the uncertainty map U output by the uncertainty head constitutes a gating term to normalize the cross-view feature aggregation weights, thereby reducing the false detection rate in occlusion and weak texture scenes.

[0033] Furthermore, the BEV grid is divided into polar coordinate rings with equal angular resolution in the near field region and into regular Cartesian grids in the far field region, and the feature energy is kept continuous through differentiable resampling in the boundary region between the two domains.

[0034] Furthermore, the CenterPoint detection head includes a heatmap branch, an offset branch, a size branch, an orientation branch, and a height branch. The height branch generates a height prior based on the road normal and road type, while the orientation branch adopts a two-stage angle regression and introduces inter-frame heading smoothing regularization.

[0035] Furthermore, the epipolar consistency loss is based on the multi-view epipolar geometry relationship constraint to ensure the consistency of the feature response of the same BEV unit in each view, thereby improving the cross-view registration accuracy and detection stability.

[0036] Furthermore, a course-based learning strategy is adopted during the training phase: in the first phase, the self-calibration and uncertainty modules are frozen, and only the detection-related loss is minimized; in the second phase, all modules are unfrozen and epipolar consistency and extrinsic sparsity regularization are introduced to prevent instability in the optimization process.

[0037] The present invention also provides a method for detecting targets by multi-view fusion outside a vehicle, comprising the following steps;

[0038] Step 1: Obtain the multi-view image dataset of the autonomous vehicle and preprocess the images to obtain the multi-view image dataset; the multi-view images refer to the images captured by cameras at different positions on the vehicle.

[0039] Step 2: Extract scale features from the multi-view images;

[0040] Step 3: Construct a multi-view fusion target detection model based on bilinear sampling. The model input is the extracted fusion scale features, and the output is the 3D target detection results of the multi-view images.

[0041] Step 4: Train the multi-view fusion object detection model using the multi-view image dataset;

[0042] Step 5: Use the trained multi-view fusion object detection model to perform multi-scale object detection.

[0043] In step 1, the multi-view image dataset of the autonomous vehicle consists of images obtained by cameras in each view, namely, the front-facing camera, the left-facing camera, the right-facing camera, the left-rear-facing camera, the right-rear-facing camera, and the rear-facing camera.

[0044] Step 1 includes:

[0045] Step 1.1: Obtain the dataset of the front, left front, right front, left rear, right rear, and rear cameras of autonomous vehicles from the public dataset;

[0046] Step 1.2: Obtain the camera intrinsic parameter matrix specified in the public dataset. The camera intrinsic parameter matrix includes a rotation matrix and an offset matrix; these are used for calculating the fixed parameters of the geometric mapping in step 3 using the bilinear sampling method.

[0047] Step 1.3: Preprocess the multi-view images in the public dataset, including rotating and translating the images obtained from the public dataset, for use in multi-view feature extraction in Step 2. The preprocessed data will enhance the extraction effect of the SA_ResNet101 network in Step 2.

[0048] These preprocessed images are then fed into the SA_ResNet101 backbone network in step 2 for multi-view feature extraction.

[0049] Step 2 includes:

[0050] Step 2.1: Use SA_ResNet101 as the backbone network for multi-view feature extraction to extract features from the multi-view images obtained in Step 1, and obtain feature outputs at different scales;

[0051] Step 2.2: Take the outputs of the three original feature layers Conv3, Conv4, and Conv5 of different scales obtained from the SA_ResNet101 backbone network in Step 2.1 as the input of the weighted fusion feature pyramid (WSFPN);

[0052] Then, two pre-processing SFMs generate two synthetic feature layers through three original feature layers, forming a total of 5 feature layer outputs, which serve as the input to the first SFB fusion block. The SFB contains 5 SFMs, which further fuse the 5 input feature layers to obtain 5 feature layer outputs. After three SFB fusion blocks, three final original feature layers and two final synthetic feature layers are obtained. Each SFM performs linear upsampling and linear downsampling on the features input from the upper and lower channels at different scales. Then, the upsampled and downsampled features are added element-wise with the intermediate input features and passed through a 3×3 convolutional network to obtain a feature layer output by an SFM, resulting in three final original feature layers and two final synthetic feature layers.

[0053] Step 2.3: Perform a 1×1 convolution operation on the outputs of the generated three original layers and two synthetic layers, and then use the Sigmoid activation function to weight the features after the 1×1 convolution to form corresponding weights;

[0054] Step 2.4: The weighted features from Step 2.4 are fused along the channel dimension using the channel splicing module to form a joint feature representation containing multi-scale semantics. Then, a unified multi-view fusion feature map is obtained by using 1×1 convolution again.

[0055] The SA_ResNet101 processing in step 2.1 includes the following steps:

[0056] Step 2.1.1: Input the multi-view camera images obtained in Step 1 into the SA_ResNet101 backbone network;

[0057] Step 2.1.2: The SA_ResNet101 network extracts Conv3, Conv4, and Conv5 features from the image step by step through multi-layer convolution operations. The output feature map of each layer is adaptively weighted with the feature maps of other layers. The attention weight of each feature is calculated and the features are weighted and adjusted through a self-attention mechanism to form weighted and fused features of different scales.

[0058] The SFB processing procedure in step 2.2 includes:

[0059] Step 2.2.1: SFB consists of several fusion sub-modules (SFM). The features of different scales obtained in step 2.1 will first pass through the SFM module to perform upsampling and downsampling operations on the original features, and then synthesize them with the original features to obtain features of other scales.

[0060] Step 2.2.2: SFM will merge the features of the input scale to obtain features of other scales, upsample and downsample the input scale features to generate new scales, add them to the features of the original scale, and then pass them through a convolutional network to obtain new scale features.

[0061] Step 3 includes:

[0062] Step 3.1: Construct the mapping relationship from the world coordinate system to the BEV space. Based on the camera's imaging principle, derive the mapping relationship between the image coordinate system and the BEV space coordinate system:

[0063]

[0064] in These are pixel coordinates in the image coordinate system. In camera coordinate system coordinate, It is the camera's intrinsic parameter matrix. It is the extrinsic parameter matrix of the camera. Let be a rotation matrix. Let be the camera translation vector. BEV spatial coordinates;

[0065] Step 3.2: Using the mapping relationship between the image coordinate system and the BEV space from Step 3.1, map the voxel feature positions in the BEV space to the corresponding image coordinate system, and then obtain the features of the corresponding voxel positions using the bilinear sampling method; let the pixel coordinate system have four adjacent pixels. , , , Their corresponding eigenvalues ​​are respectively , , , For sub-pixel coordinates as Its corresponding eigenvalue is ;

[0066] First, calculate the horizontal interpolation:

[0067]

[0068]

[0069] Next, calculate the interpolation formula in the vertical direction:

[0070]

[0071] Finally, the feature values ​​of the corresponding sub-pixel positions are obtained by using the geometric projection relationship of BEV space voxels, forming BEV space features;

[0072] Step 3.3: After obtaining the BEV spatial features in Step 3.2, multi-target detection is performed using the CenterPoint detection head. The corresponding loss function is designed as follows:

[0073]

[0074] in For heatmap loss, For offset loss, For scale loss, Towards loss, To highly predict losses.

[0075] The training objective of step 4 is average accuracy.

[0076] The average precision refers to the area under the Precision-Recall curve for each target category. ), for all categories Take the average value to get That is, the model must satisfy the target relation as follows:

[0077]

[0078] in It is the total number of target categories. It is the average precision of the c-th category.

[0079] A system for implementing a multi-view fusion target detection method for vehicle exteriors includes a data preprocessing module, a multi-view image feature extraction module, a BEV3D feature generation module, and a 3D target detection module.

[0080] The data preprocessing module is used to obtain the camera's extrinsic and intrinsic parameter matrices and to perform rotation operations on the images acquired by the camera.

[0081] The multi-view image feature extraction module includes a backbone network SA_ResNet101 and a WSFPN network. The SA_ResNet101 network performs the first round of feature extraction on the multi-view image after the data preprocessing module to obtain features at each scale. The WSFPN network is used to perform weighted fusion processing on the features at each scale to obtain more detailed scale features.

[0082] The BEV3D feature generation module uses the relationship between image coordinates and world coordinates to obtain the geometric mapping relationship between image coordinates and BEV space coordinates, and then obtains the corresponding features in BEV space according to the bilinear sampling method.

[0083] The 3D target detection module uses the CenterPoint detection head to perform target detection on BEV spatial features.

[0084] The proposed multi-view fusion target detection method for vehicle exteriors is used in autonomous driving, visual perception, and target detection.

[0085] The beneficial effects of this invention are:

[0086] This invention utilizes a backbone network with an attention mechanism to better perform the first-layer scale feature extraction, thereby enhancing scene representation. By leveraging WSFPN, it overcomes the shortcomings of uneven scale interval distribution in traditional multi-scale feature extraction methods. By mapping the BEV space using bilinear sampling, it significantly reduces the model's training parameters. Specifically, the training parameters 44.551M are derived solely from the backbone network's training parameters, replacing deep learning training methods, thus saving computational resources and reducing training costs.

[0087] The technology of this invention enables the deep fusion of previously independent scale features through a spatial attention mechanism, resulting in rich feature information. This allows for more accurate, stable, robust, and lightweight target detection, which has broad application prospects in the field of autonomous driving. Attached Figure Description

[0088] Figure 1 This is a schematic diagram of the vehicle exterior multi-view fusion target detection method provided in an embodiment of the present invention.

[0089] Figure 2 This is a flowchart of the vehicle exterior multi-view fusion target detection method provided in the embodiments of the present invention.

[0090] Figure 3 This is a WSFPN network structure diagram provided in an embodiment of the present invention.

[0091] Figure 4 This is a schematic diagram of the bilinear sampling method set mapping provided in an embodiment of the present invention.

[0092] Figure 5 This is a schematic diagram illustrating the visualization results of the model on the test set, provided by an embodiment of the present invention. Detailed Implementation

[0093] The present invention will now be described in further detail with reference to the accompanying drawings.

[0094] In existing visual perception technologies for autonomous vehicles, images from multiple cameras are typically processed independently, making it difficult to achieve effective alignment of spatial relationships. This is especially problematic in 3D detection tasks, where issues such as blurred target boundaries, positional shifts, and scale inconsistencies easily arise. This directly leads to blind spots when the vehicle perceives its external environment, impacting the safety of obstacle avoidance, lane changing, and path planning. The core technical challenge for industrial applications lies in unifying image information from cameras at different locations into a single spatial representation, enabling the system to accurately obtain 3D position and shape information, thereby improving the stability and robustness of detection.

[0095] To address the aforementioned issues, the proposed multi-view fusion object detection method for vehicle exteriors establishes a mapping relationship from the camera coordinate system to the bird's-eye view space, projecting 2D image features from different perspectives into a unified 3D space. In this process, bilinear sampling interpolation is used to achieve pixel-level feature alignment, enabling continuous spatial transition of feature information from different cameras and avoiding information fragmentation. Through this mechanism, spatial points around the vehicle can obtain clear voxel representations within the BEV space, overcoming the shortcomings of existing technologies in achieving 3D geometric unification.

[0096] In the feature extraction stage, a deep convolutional backbone network based on a self-attention mechanism is employed to transform the original multi-view images into multi-scale feature representations. This network not only captures local texture features but also enhances information across channels and scales through attention weights, thus maintaining information consistency during feature fusion at different resolutions. This method effectively solves the problems of small target loss and insufficient detail in large targets that traditional convolutional networks often encounter when processing multi-scale objects.

[0097] In the feature fusion stage, a multi-level fusion module is introduced, achieving dynamic compensation between features of different scales through alternating upsampling and downsampling operations. This multi-level feature reconstruction mechanism enables the final fused features to simultaneously preserve the geometric details of small objects at close range and the global contours of large objects at long distances. Compared to existing single-scale feature fusion methods, this method has significant advantages in semantic representation and geometric consistency, ensuring the stability of detection results under various operating conditions.

[0098] In the 3D detection process, a CenterPoint-based detection head is employed, inputting BEV spatial features into the detection network. A multi-task joint optimization objective is constructed by combining heatmap loss, offset loss, scale loss, orientation loss, and height loss. This design can simultaneously learn the target's center position, geometric dimensions, orientation angle, and vertical height information, ensuring the completeness and accuracy of the 3D detection results. This multi-objective joint learning approach solves the bias problem caused by training with a single loss function in existing methods, making the detection model more robust in complex traffic environments.

[0099] Through the above methods, the requirement for global perception of the external environment in autonomous driving is ultimately met in industrial applications. After unified geometric mapping and feature fusion, multi-view images can form a stable and reliable 3D semantic representation, thereby improving the vehicle's environmental understanding capabilities in urban roads, highways, and complex intersection scenarios. This not only solves the technical bottlenecks of traditional methods in detection accuracy and spatial consistency, but also provides a low-latency, high-precision, and highly adaptable solution for industrial deployment.

[0100] like Figure 1 , Figure 2 As shown, a multi-view fusion target detection method for vehicle exterior includes the following steps:

[0101] Step 1: The nuScenes public dataset is one of the most authoritative datasets in the field of autonomous driving. It has a large training set to help with testing. First, obtain images from different perspectives in the nuScenes public dataset and preprocess the images to obtain a multi-view image dataset.

[0102] In this embodiment, step 1 includes:

[0103] Step 1.1: In this embodiment, first obtain the image datasets of the front left, front right, front front, rear left, rear right, and rear cameras from the nuScenes public dataset;

[0104] Step 1.2: Obtain the camera intrinsic and extrinsic parameters (rotation matrix and offset matrix) published in the nuScenes public dataset.

[0105] Step 1.3: Rotate the obtained multi-view images and add them to the training dataset to complete data preprocessing;

[0106] Step 2: Perform preliminary feature extraction on the multi-view images to obtain preliminary features at each scale. By using a backbone network with an attention mechanism, the first-layer scale feature extraction can be completed more effectively, enhancing scene representation.

[0107] Specifically, step 2 includes:

[0108] Step 2.1: Using SA_ResNet101 as the backbone network, the multi-view image features are first processed through the original SA_ResNet101 to obtain unweighted feature maps;

[0109] Step 2.2: By pooling the feature map without weights, a two-dimensional spatial attention map is generated, which represents the relative importance of different spatial locations in the image. Then, this attention map is used to generate the final spatial attention weights through operations such as convolution.

[0110] Step 2.3: After obtaining the spatial attention weights, the input feature map is weighted. That is, these weights are used to enhance key information regions in important parts of the image.

[0111] Step 2.4: Use the final Conv3, Conv4, and Conv5 layer features as the output features of SA_ResNet101.

[0112] Step 3: Construct a multi-view fusion target detection model for vehicle exterior, inputting the output features obtained in Step 2. By utilizing WSFPN, the shortcomings of traditional multi-scale feature extraction methods, such as uneven distribution of scale intervals, can be overcome.

[0113] In this embodiment, step 3 includes:

[0114] Step 3.1: Construct an SFM feature fusion block, combining features from different layers. (The i-th layer features) are input into the SFM. The SFM includes a linear scaling module (using upsampling or downsampling) and a 3×3 convolution module, which uses upsampling... and downsampling The operation can obtain features at different scales, and then the linearly scaled feature maps are stacked pixel by pixel to initially fuse the features at different scales. Finally, features are further extracted and information is fused using 3×3 convolution to obtain... 3×3 convolution can refine and fuse features without changing the scale.

[0115] Step 3.2: Construct the SFB feature fusion block. The SFB consists of 5 SFMs, such as... Figure 3 The SFB shown uses 5 SFMs to obtain five feature outputs at different scales. It consists of three original layers and two fusion layers, which makes the scale feature size of the output not too discontinuous, thus obtaining richer feature information.

[0116] Step 3.3: Concatenate the three SFB feature fusion blocks together to obtain the final multi-scale fusion feature.

[0117] Step 3.4: The five output features at different scales are sequentially passed through a Conv1×1 convolutional network and a Sigmoid activation function to obtain initial weights, which are then used to weight the features at each scale. to Subsequently, features at different scales are upsampled and weighted for fusion. These features are then fused along the channel dimension using a channel stitching module to form a joint feature representation containing multi-scale semantics. Finally, to integrate the stitched feature information and compress the channel dimension, we again use 1×1 convolution for feature mapping to obtain a unified multi-view fusion feature map. This feature will be used in subsequent BEV space mapping and 3D object detection modules.

[0118] Step 3.5: As Figure 4 As shown, BEV space coordinates are mapped to the image coordinate system according to the geometric correspondence between image coordinates and BEV space coordinates. Then, bilinear sampling is used to obtain the multi-view features of the corresponding coordinates. Mapping the BEV space using bilinear sampling can significantly reduce the training parameters of the model, saving computational resources compared to deep learning training methods.

[0119] The mapping relationship between image coordinates and BEV space is as follows:

[0120]

[0121] in These are pixel coordinates in the image coordinate system. In camera coordinate system coordinate, It is the camera's intrinsic parameter matrix. It is the extrinsic parameter matrix of the camera. Let be a rotation matrix. Let be the camera translation vector. These are the coordinates in BEV space.

[0122] The expression for the bilinear sampling method is as follows:

[0123]

[0124] in represent The mapping feature values, These are the sub-pixel coordinates.

[0125] Step 3.6: Use the CenterPoint detector head as the detector head for the model.

[0126] Step 4: Train the multi-view fusion object detection model for vehicle exterior using the nuScenes dataset;

[0127] The training objective is average accuracy.

[0128] The average precision refers to the area under the Precision-Recall curve for each target category. ), for all categories Take the average value to get That is, the model must satisfy the target relation as follows:

[0129]

[0130] in It is the total number of target categories. It is the average precision of the c-th category.

[0131] The loss function corresponding to the model is:

[0132]

[0133] in For heatmap loss, For offset loss, For scale loss, Towards loss, To highly predict losses.

[0134] by The minimum value is the training objective, and the initial learning rate is set to . The specific training methods and related parameters are similar to those of mainstream training methods, and will not be elaborated here.

[0135] Step 5: Use the trained multi-view fusion target detection model to perform target detection.

[0136] It should be noted that before using the trained multi-view fusion object detection model for vehicle exterior to perform object detection, the obtained dataset needs to undergo the series of preprocessing steps described above.

[0137] Table 1 shows the experimental results of the basic method on the nuScenes dataset.

[0138] Evaluation metrics mATE mASE mAOE mAVE mAAE mAP Scores 0.648 0.270 0.410 0.430 0.270 0.407

[0139] Wherein, mATE measures the average translation error between the detected target and the ground truth target, mASE measures the average scale error between the detected bounding box and the ground truth box, mAOE measures the orientation error between the detected bounding box and the ground truth box, mAVE measures the velocity prediction error between the detected target and the ground truth target, mAAE measures the acceleration error between the detected target and the ground truth target, and mAP measures the average accuracy of the model across all categories and all detection tasks.

[0140] Table 2 Comparison of the clumsy method with other state-of-the-art work on the nuScenes dataset.

[0141]

[0142] As shown in Tables 1 and 2, this embodiment uses the test set obtained by processing the nuScenes dataset for experiments. The experimental results demonstrate that the model achieves an mAP score of 0.407 and an NDS score of 0.501 on the nuScenes dataset. This surpasses the detection accuracy of then-current state-of-the-art multi-object detection methods while using fewer parameters. (See attached table.) Figure 5 The image shows the detection performance visualization interface of the model in this application. Almost all targets can be detected by the model in this application, including occluded targets, dense scene targets, and distant targets.

[0143] Through the above steps, the method of the present invention can utilize the WSFPN structure to compensate for the incomplete capture of feature information caused by the large scale difference in traditional methods, and greatly reduce the model parameters mapped to the BEV space through bilinear sampling, which is almost parameterless compared to the mapping process based on the Transformer method.

[0144] Example 1

[0145] A self-driving test vehicle is equipped with wide-angle cameras in six directions (front, rear, left, and right) to collect multi-view image data of urban road scenes. The raw multi-view image data does not require geometric alignment and is directly used as input data. After noise reduction processing by a preprocessing module, a standardized multi-view image set is formed.

[0146] Based on this, the image is input into a subsequent feature extraction and detection model, realizing a complete process of fusing views from cameras in different directions into a unified view to complete target detection. This method ensures the accuracy of spatial data fusion and effectively supports the implementation of the overall detection framework.

[0147] Example 2

[0148] The multi-view data obtained in Example 1 is input into the SA_ResNet101 network, and features from three layers (Conv3, Conv4, and Conv5) are extracted layer by layer. Each layer's features are weighted through an attention mechanism to obtain an enhanced multi-scale feature representation.

[0149] The three-layer features are then input into the weighted fusion feature pyramid (WSFPN), where they are upsampled and downsampled in the SFM module and synthesized with the intermediate features. Three SFB fusion blocks output three layers of original features and two layers of synthesized features. Finally, the fused joint feature representation is obtained by channel concatenation, effectively improving cross-scale detection accuracy.

[0150] Example 3

[0151] When constructing the mapping between image coordinates and BEV space, the projection relationship is derived using the intrinsic and extrinsic parameter matrices of the vehicle camera, mapping pixels in the image to voxel positions in BEV space. For each point in BEV space, the corresponding eigenvalue is calculated from the four neighborhoods of the image pixel using a bilinear sampling formula.

[0152] For example, for the BEV voxel position of a target vehicle in the area in front of the vehicle, after mapping it to image pixels, smooth feature values ​​are obtained through horizontal and vertical interpolation formulas. The BEV features generated in this way avoid the feature fragmentation problem and maintain the continuity of spatial projection.

[0153] Example 4

[0154] The BEV features obtained in Example 3 are input into the CenterPoint detector head to predict multiple types of 3D targets, including vehicles, pedestrians, and cyclists. The loss function is designed as L_total, which includes heatmap loss, offset loss, scale loss, orientation loss, and height loss, each of which constrains different output parameters of the model.

[0155] During training, taking the public dataset nuScenes as an example, the Precision-Recall curves for various targets were calculated, and the mean accuracy (mAP) was used as the training convergence target. Experimental results show that the proposed detection method maintains high accuracy even in scenarios with dense lane lines and occlusion, verifying the effectiveness of the proposed method.

[0156] Example 5

[0157] A vehicle-mounted multi-view fusion detection system was constructed, comprising a data preprocessing module, a feature extraction module, a BEV feature generation module, and a detection module. The preprocessing module is responsible for correcting images acquired by cameras. The feature extraction module extracts multi-scale features based on SA_ResNet101 and WSFPN. The BEV module implements geometric mapping and bilinear sampling. The detection module outputs 3D detection results based on the CenterPoint detection head.

[0158] Deployed on an onboard computing platform and equipped with GPU-accelerated inference, the system achieves real-time processing of 20 frames per second in urban road tests. Experimental results show that the system can operate stably under different lighting and weather conditions, meeting the real-time and accuracy requirements of autonomous driving applications.

[0159] Example 6

[0160] The above methods and systems can be applied to autonomous vehicles to achieve three-dimensional perception of the surrounding environment. When the vehicle is driving on multi-lane urban roads, the system can simultaneously detect vehicles in front, pedestrians on the side, and cyclists behind, forming a complete perception map of the external environment.

[0161] Based on the detection results, the autonomous driving decision-making module can accurately execute actions such as lane changing, obstacle avoidance, and emergency braking. This application verifies the industrial value of multi-view fusion object detection methods in real traffic environments, contributing to improved safety and reliability of autonomous driving.

[0162] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A method for off-board multi-view fusion three-dimensional object detection, characterized in that, The method comprises the following steps: S0, cross-camera time delay estimation and compensation based on camera timestamp and speedometer / IMU information, to obtain a time-aligned multi-view image sequence; S1, obtaining and preprocessing a multi-view image dataset of an autonomous vehicle; S2, incrementally updating the extrinsic parameters of each view based on an online extrinsic self-calibration module, the self-calibration being constrained by lane lines, road plane and short-time visual odometry, and optimizing SE(3) disturbance within a sliding window; S3, extracting multi-scale features from each view image using a backbone network with a spatial attention mechanism, and compensating for feature distortion caused by rolling shutter and vehicle motion through a time domain adapter; S4, using adaptive four-point sampling (AFQS) to perform continuous sampling of a learnable anisotropic kernel in the pixel sub-domain, combining visibility estimation and uncertainty weight to complete cross-view fusion, and generating polar-Cartesian hybrid BEV feature grid; S5, inputting the BEV features into a CenterPoint detection head with height prior and heading continuity regularization, and outputting the center, size, height and orientation of the three-dimensional target; S6, end-to-end training of the network based on a joint loss composed of heat map loss, offset loss, size loss, orientation loss, height loss, polar line consistency loss, uncertainty entropy regularization and extrinsic parameter sparsity regularization.

2. The method of claim 1, wherein, The extrinsic self-calibration of S2 is performed by homographic constraint fitting of common lane line features between multi-view frames, combining short-time pose information calculated by IMU, and using sparse Levenberg-Marquardt optimization to obtain extrinsic parameter increment Δξ, and updating the initial extrinsic parameter matrix in Lie algebra form.

3. The method of claim 1, wherein, The adaptive four-point sampling (AFQS) obtains pixel coordinates (u, v) by back-projection for each BEV unit, outputs four sub-displacements {Δu_i, Δv_i} from the offset prediction head, and performs weighted aggregation with a learnable elliptical anisotropic kernel Kθ, thereby realizing high-precision feature fusion in continuous space.

4. The method of claim 1, wherein, The visibility estimation generates an occlusion probability map M through a differentiable Z-buffer, and together with the uncertainty map U output by the uncertainty head, forms a gating term to normalize the cross-view feature aggregation weight, thereby reducing the false detection rate in occlusion and weak texture scenes.

5. The method of claim 1, wherein, The BEV grid is divided into polar coordinate ring bands with equal angular resolution in the near-field region, and into regular Cartesian grid in the far-field region, and through differentiable resampling, the feature energy is kept continuous in the boundary region between the two domains.

6. The method of claim 1, wherein, The CenterPoint detection head includes a heat map branch, an offset branch, a size branch, an orientation branch and a height branch, wherein the height branch generates a height prior based on the road normal and the road type, and the orientation branch uses two-stage angle regression and introduces inter-frame heading smoothing regularization.

7. The method of claim 1, wherein, The polar line consistency loss is based on the constraint of the feature response consistency of the same BEV unit in each view under the multi-view polar geometric relationship, thereby improving the cross-view registration accuracy and detection stability.

8. The method of claim 1, wherein, The training stage adopts a curriculum learning strategy: in the first stage, the self-calibration and uncertainty module is frozen, and only the detection-related loss is minimized; in the second stage, all modules are unfrozen and the epipolar consistency and sparse regularization of the external parameter are introduced to prevent the optimization process from being unstable.

9. An off-board multi-view fusion object detection system, comprising: Comprise: a time synchronization and external parameter self-calibration module for performing S0 and S2 of claim 1; a multi-scale feature extraction module for performing S3; an AFQS fusion and visibility gating module for performing S4; a hybrid BEV construction module for generating a grid from the output of S4; a three-dimensional detection module for performing S5 and outputting target parameters; a joint loss training module for performing S6.

10. A computer readable storage medium having stored thereon a computer program which, when executed by a processor, performs the method steps of any one of claims 1 to 8.

Images