Automatic driving-oriented multi-sensor fused 3D target detection method and medium

Abstract

The invention relates to an automatic driving-oriented multi-sensor-fused 3D target detection method and a medium. The method comprises the following steps: acquiring a vehicle pose and point clouds and images of a vehicle surrounding scene in real time; carrying out image enhancement processing by adopting a dark light enhancement network, and carrying out image feature extraction to obtain a multi-view feature map; performing view transformation based on the multi-view feature map to obtain an initial camera BEV feature, and performing updating by fusing the vehicle pose with the multi-frame camera BEV feature to obtain a final camera BEV feature; carrying out sampling processing on the final camera BEV features to obtain multi-scale camera BEV features, and carrying out feature extraction on the point cloud to obtain multi-scale LiDAR BEV features; performing feature fusion to obtain fused features; and based on the fusion features, performing three-dimensional target identification by using a 3D target detection model, and outputting a 3D object detection frame. Compared with the prior art, the method has the advantages of improving 3D detection performance and the like.

Description

Technical Field

[0001] The present invention relates to the field of target detection technology, and in particular to a 3D target detection method and medium for autonomous driving that integrates multiple sensors. Background Art

[0002] 3D object detection is a task that predicts the properties of an object in 3D space by inputting sensor data. The object to be detected in 3D space can be defined as a cuboid. The cuboid uses (x, y, z) to represent the center coordinates of the object, (l, w, h) to represent the length, width and height of the object, θ to represent the deflection angle (the deflection angle of the cuboid on the ground plane), and class to represent the category of the target object. As an important part of the autonomous driving perception system, 3D object detection can help vehicles better understand the surrounding environment and make safer and more efficient driving decisions.

[0003] Autonomous vehicles are usually equipped with multiple sensors, including cameras, LiDAR (laser radar), Radar (millimeter wave radar), GPS (global positioning system) and IMU (inertial measurement unit) to obtain data such as images, point clouds, and vehicle posture. In the field of 3D target detection, there are certain defects in target detection through a single sensor, and it is impossible to fully obtain complete object information. Specifically, images can provide rich semantic information but cannot provide location information of objects in space, and they are more sensitive to lighting conditions. Point clouds can reflect precise structural location information, but their data distribution is relatively sparse, and they will have more noise points in bad weather.

[0004] 3D target detection based on multiple sensors can complement the features of various forms of data to improve the detection performance of the model. Although multimodal fusion can often achieve good detection results, due to the heterogeneity of data from different sensors, the alignment of different modal features needs to be considered during multi-sensor fusion. In addition, in actual scenarios, the algorithm needs to be able to cope with various scenarios, such as darkness, sensor damage and loss, etc. In these scenarios, the overall perception ability will be reduced due to the limitations of each sensor itself.

[0005] In the 3D object detection task on the autonomous driving vehicle side, in order to improve the accuracy and robustness of 3D object detection, it is necessary to extract features from data from different sensors and align the features in a unified high-dimensional space. In addition, it is necessary to alleviate the problem that the model's detection performance deteriorates severely in low light and when LiDAR is missing. Summary of the invention

[0006] The purpose of the present invention is to provide a 3D target detection method and medium for autonomous driving that integrates multiple sensors and improves the accuracy and robustness of 3D target detection.

[0007] The purpose of the present invention can be achieved by the following technical solutions:

[0008] A 3D target detection method for autonomous driving by integrating multiple sensors includes the following steps:

[0009] Collecting the vehicle posture and the point cloud P and image I of the scene around the vehicle in real time, wherein the point cloud, image and vehicle posture are collected by different sensors;

[0010] Based on the image I, a dark light enhancement network is used to perform enhancement processing, and the enhanced image Perform image feature extraction to obtain multi-view feature maps

[0011] Based on the multi-view feature map Performing view transformation to obtain initial camera BEV features, and using the vehicle posture to fuse multiple frames of camera BEV features for updating to obtain final camera BEV features;

[0012] The final camera BEV feature is sampled to obtain a multi-scale camera BEV feature And extract features from the point cloud to obtain multi-scale LiDAR BEV features

[0013] The multi-scale camera BEV features and multi-scale LiDAR BEV features Perform feature fusion to obtain fusion feature F fused ;

[0014] Based on the fusion feature F fused , the 3D target detection model is used to perform three-dimensional target recognition, and the 3D object detection box is output to complete the detection process.

[0015] Furthermore, the step of performing enhancement processing on the image I comprises:

[0016] The image I captured by camera c c Classified as daytime images and night images

[0017] If it is a daytime image No enhancement processing is performed;

[0018] If it is a night image An unsupervised dark light enhancement network is used Enhance the image brightness to obtain an enhanced night image The dark light enhancement network is an EnlightenGAN network, and the enhanced night image It is expressed as:

[0019]

[0020] Repeat the above steps for the images collected by each camera, and then collect all the daytime images. and all enhanced night images Take the union and get the final enhanced image The enhanced image It is expressed as:

[0021]

[0022] In the formula, ∪ represents the union operation.

[0023] Furthermore, the multi-view feature map is obtained The steps include:

[0024] Based on the enhanced image An image encoder is used to extract features to obtain a multi-scale feature map, wherein a Swin Transformer is used as a backbone network of the image encoder;

[0025] A feature pyramid network is used to aggregate the multi-scale feature maps in a cascade manner to obtain a multi-view feature map.

[0026] Furthermore, the step of obtaining the initial camera BEV feature includes:

[0027] For the multi-view feature map The pixel (u,v) in defines the discrete depth d of the pixel (u,v). i =d0+i*δd,i∈{0,1,…,|N d |}, where d0 is the initial depth, |N d | is the number of discrete depths, δd is the depth interval;

[0028] For each discrete depth d i , the camera parameter K is used to convert the pixel (u, v) into a point cloud in 3D space, where the conversion expression is:

[0029]

[0030] Where (x, y, z) is the point cloud coordinate in 3D space;

[0031] For the multi-view feature map Perform the above steps for each pixel in to obtain the frustum in 3D space;

[0032] Based on the multi-view feature map Use the depth estimation network to obtain the depth distribution feature map and contextual features The depth distribution feature map and contextual features Perform outer product on the channel dimension to obtain the weighted feature map

[0033] The weighted feature map The features in are assigned to the point cloud in the frustum Frustrum to obtain the frustum feature G;

[0034] The 3D space is voxelized, the point cloud features in each voxel in the cone feature G are aggregated, and pooling is performed along the Z axis to compress the point cloud features to the BEV plane to form an initial camera BEV feature.

[0035] Furthermore, the step of obtaining the final camera BEV feature includes:

[0036] Based on the vehicle posture, the posture transformation matrix of the vehicle from time t-1 to time t is obtained The pose transformation matrix It is expressed as:

[0037]

[0038] In the formula, represents the rotation matrix, represents the translation matrix;

[0039] Based on the pose transformation matrix The camera BEV feature at time t-1 is transformed by matrix Convert to the BEV coordinate system at the current time t to obtain the BEV features of the camera aligned at time t-1 During the conversion process, the translation vector of the transformation matrix is Multiply by the scale factor between real world and BEV cells;

[0040] The BEV features of the camera aligned at time t-1 and initial camera BEV characteristics Splice and add position encoding as query, and use the camera BEV features aligned at time t-1 and initial camera BEV characteristics As values, respectively, the deformable attention mechanism is used for calculation to obtain and in and The expression is:

[0041]

[0042] Where DeformableAttn(*) is the deformable attention operation, [*] is the channel dimension concatenation, and Pos is the feature position encoding;

[0043] Using the fully connected layer and The mean of the camera is mapped and compared with the initial camera BEV feature The camera BEV feature at time t is summed and updated as the final camera BEV feature, wherein the final camera BEV feature is expressed as:

[0044]

[0045] In the formula, the left side of the equation is is the final camera BEV feature, the right side of the equation is the initial camera BEV feature, MLP(*) is the fully connected layer, and AVERAGE(*) is the averaging operation.

[0046] Furthermore, the multi-scale camera BEV features are obtained The steps include:

[0047] The residual module is used to transform the final camera BEV features Downsampling is performed to obtain features X1 and X2 of different resolutions, where features X1 and X2 are expressed as:

[0048]

[0049] X2=Residual(X1)

[0050] Where, Residual(*) represents the residual module;

[0051] Feature X2 is upsampled, concatenated with feature X1, and input into the residual module to obtain feature X3, where feature X3 is expressed as:

[0052] X3=Residual(Concat(X1,Upsample(X2)))

[0053] In the formula, Upsample(*) represents upsampling using bilinear interpolation, and Cocat(*) represents concatenation operation;

[0054] The final camera BEV features and feature X3 as multi-scale camera BEV features The multi-scale camera BEV features It is expressed as:

[0055]

[0056] Where l represents the multi-scale level.

[0057] Furthermore, the multi-scale LiDAR BEV features are obtained The steps include:

[0058] Limiting the range of the point cloud on the X-axis, the Y-axis and the Z-axis, and voxelizing the point cloud to obtain a voxelized point cloud;

[0059] Based on the voxelized point cloud, the VoxelNet network is used to extract point cloud features. During the point cloud feature extraction process, the VoxelNet network compresses the intermediate layer point cloud features to the BEV plane as the multi-scale LiDARBEV feature.

[0060] Furthermore, the obtained fusion feature F fused The steps include:

[0061] Define multi-scale LiDAR BEV features for scale l and multi-scale camera BEV features

[0062]

[0063] Where, X l ,Y l represents the characteristic size of BEV at scale l;

[0064] Using convolution π to improve The channel dimension of The channel dimensions of are the same;

[0065] For the reference point (x l ,y l ), and obtain the feature vectors and

[0066] Will Added to the position encoder as the query of the deformable attention mechanism, As the value of the deformable attention mechanism, the offset Δoffset(x l ,y l) and weight graph A(x l ,y l );

[0067] Based on the offset Δoffset(x l ,y l ), weight graph A(x l ,y l ) and (x l ,y l ) to obtain fusion features of different scales The fusion features of different scales It is expressed as:

[0068]

[0069] Where DeformableAttn(*) is the deformable attention, Pos is the feature position code, M is the number of attention heads, N is the number of sampling points, and W is the m and W′ m Represents a learnable mapping layer;

[0070] Based on the fusion features of different scales The bilinear interpolation method is used for further fusion to obtain the fusion features of each scale and then combined with Add them together to get the final fusion feature F fused , where the final fusion feature F fused It is expressed as:

[0071]

[0072] Where Downsample(*) represents downsampling using bilinear interpolation, and L is the number of scales used.

[0073] Furthermore, the 3D target detection model is a Transfusion-L model, which uses a DETR method to perform 3D target detection and outputs a 3D object detection frame, wherein the 3D object detection frame includes a center position, size, and yaw angle.

[0074] The present invention also provides a computer-readable storage medium, comprising one or more programs for execution by one or more processors of an electronic device, wherein the one or more programs include instructions for executing the 3D target detection method for autonomous driving using multiple sensors as described above.

[0075] Compared with the prior art, the present invention has the following beneficial effects:

[0076] (1) The present invention extracts image features from images and then fuses the camera BEV features of multiple frames to update the camera BEV features of the current frame when acquiring the camera BEV features. This can effectively solve the problem of severe degradation in detection performance when point clouds are missing, and performs 3D target detection by fusing multi-scale camera BEV features and LiDAR BEV features, which helps to improve the accuracy and robustness of 3D target detection.

[0077] (2) The present invention uses a dark light enhancement network to enhance the brightness of night images, thereby alleviating the problem of decreased detection performance due to underexposure of night images.

[0078] (3) The present invention introduces motion timing information into the image stream part of 3D target detection, uses deformable attention to model historical dynamic and static information, and obtains camera BEV features of different resolutions through multi-scale downsampling, thereby improving the 3D detection performance of the model when only image input is used.

[0079] (4) The present invention uses multi-scale dual-stream (camera-LiDAR) BEV feature fusion to enable features of different modalities to be aligned in a high-dimensional BEV space. The fused features contain multi-granular spatial information, which is more conducive to the modeling and environmental perception of 3D space. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 It is a schematic diagram of the method flow of the present invention;

[0081] Figure 2 is the image enhancement flow chart of the present invention;

[0082] Figure 3 This is a network structure diagram of the camera BEV feature downsampling of the present invention;

[0083] Figure 4 This is a network structure diagram of the multimodal BEV feature fusion module of the present invention. DETAILED DESCRIPTION

[0084] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0085] This embodiment provides a 3D target detection method for autonomous driving by integrating multiple sensors. Figure 1 As shown, the method comprises the following steps:

[0086] Step S1: Install LiDAR, camera, IMU and GPS on the vehicle, and perform time synchronization and calibration on different sensors.

[0087] Install LiDAR on the top of the vehicle. Install cameras at the front, front left, front right, rear, rear left, and rear right of the vehicle, and use GNSS and IMU to achieve accurate global attitude estimation. After time synchronization of different sensors, the rotation and translation matrices between different sensors are obtained through calibration algorithms;

[0088] Step S2: Obtain point cloud, image and vehicle pose data through multiple sensors.

[0089] The 3D point cloud data P of the surrounding scene is captured in real time through the 32-line LiDAR. The coordinate position, reflection intensity and timestamp of the point cloud are retained. The point cloud captured at time t is as follows:

[0090]

[0091] Where (x i ,y i ,z i ) represents the spatial coordinates of the point cloud, f i Represents the features of the point cloud (reflection intensity, timestamp), N L Represents the number of point clouds in a frame.

[0092] The RGB image data I is acquired in real time by the surround camera. The resolution of the acquired image is 1600×900, which fully covers the 360° horizontal field of view. The image data captured at time t is as follows:

[0093]

[0094] Where N C is the number of cameras used. C is 6.

[0095] The vehicle's motion state information is obtained through GPS and IMU, including speed, acceleration, angular velocity, body coordinates in the global coordinate system, and the current timestamp. Based on the information obtained, the vehicle's posture transformation matrix from t-1 to t is calculated

[0096]

[0097] in represents the rotation matrix, Represents the translation matrix.

[0098] Step S3: Process the input image and use an enhancement network to improve the quality of the dark light image.

[0099] Adjust the resolution of the input image. The present invention downsamples the resolution to 1 / 4 of the original image. Figure 2As shown in Figure 1, the dark light enhancement network is used to improve the brightness of the underexposed areas in the night image, and finally merged with the daytime image to obtain the processed image. Wherein H and W are the length and width of the image. In the present invention, H and W are 800 and 448 respectively.

[0100] The process of enhancing the input image is as follows:

[0101] 1. For camera c, input image I c Classified as daytime images and night images

[0102] 2. Yes No processing is performed.

[0103] 3. Yes Using unsupervised dark light enhancement network Improve the brightness of the image to obtain an enhanced image EnlightenGAN is used in this invention.

[0104]

[0105] 4. Repeat the above steps for each camera c, and finally take the union of all daytime images and processed nighttime images to obtain the enhanced image

[0106]

[0107] Step S4: Use the feature extraction module to generate a multi-view feature map of the image.

[0108] Use image encoder and feature pyramid network to enhance the image Perform feature extraction to obtain the multi-view feature map of the image at time t The present invention uses Swin Transformer as the encoder backbone network to extract features and obtain multi-scale feature maps. Then, the feature pyramid network is used to aggregate feature maps of different scales in a cascade manner to output the multi-view feature map at the current time t. Among them C I is the number of channels of the feature map. I is 256.

[0109] Step S5: Use the view transformation algorithm to convert the multi-view feature map into a BEV feature map.

[0110] Input camera parameter matrix K and multi-view feature map The camera BEV feature at time t is obtained using the view transformation algorithm LSS The algorithm flow of LSS is as follows:

[0111] 1. For the input image feature map The pixel (u,v) in defines its discrete depth d i =d0+i*δd,i∈{0,1,…,|N d |}. Where d0 is the initial depth, |N d | is the number of discrete depths, and δd is the depth interval. For each depth d i , using the camera parameters K∈R 3×4 , the image (u, v) can be lifted to 3D space to obtain a point cloud. In the present invention, d0 is 4, δd is 1, and N d is 45. The conversion formula is as follows:

[0112]

[0113] Where (x, y, z) is the coordinate of the point cloud in space. Performing the above operation on each pixel in the feature map can obtain the viewing cone in space

[0114] 2. Input feature map at time t Use the depth estimation network to obtain the depth distribution feature map and contextual features The weighted feature map is obtained by performing the outer product of the two features in the channel dimension.

[0115] 3. Assign the features in W to the point cloud in Frustrum to obtain the frustum feature G. Then voxelize the space and aggregate the point cloud features in each voxel in G. Finally, perform a pooling operation along the Z axis to compress the features to the BEV plane to form the BEV feature of the camera.

[0116] Step S6: Use the attention mechanism to fuse the camera BEV features of the current frame and the historical frames.

[0117] Using the ego-vehicle pose transformation matrix The camera BEV feature at time t-1 Align with the BEV features of the current moment, and then use the deformable attention mechanism to interact the BEV features of the current frame with the BEV features of the historical frames. The specific process is as follows:

[0118] 1. Input and Through matrix transformation Convert to the BEV coordinate system at the current time t to obtain The BEV coordinate system takes the center of the BEV feature map as its origin. During the transformation process, the translation vector in the transformation matrix The ratio coefficient between the real world and the BEV cell needs to be multiplied. In the present invention, this coefficient is 0.3.

[0119] 2. Input the aligned BEV features at time t-1 and the BEV characteristics at time t Will and After concatenation, add the position code as the query. and As values, use deformable attention output and Then, the fully connected layer is used and The mean of is mapped and finally The sum is performed to update the BEV features of the current frame. The fusion process is as follows:

[0120]

[0121] Among them, DeformableAttn(*) is deformable attention, [*] is channel dimension concatenation, Pos is feature position encoding, MLP(*) is the fully connected layer, and AVERAGE(*) is the averaging operation.

[0122] Step S7: down-sample the camera BEV features of the current frame to obtain multi-scale features.

[0123] like Figure 3 As shown, the residual module Residual is used to calculate the BEV feature map at the current moment Downsample to obtain features X1 and X2 of different resolutions. Then upsample X2, concatenate it with X1 and input Residual to obtain X3. and X3 as multi-scale camera BEV features The overall process is as follows:

[0124]

[0125] X2=Residual(X1)

[0126] X3=Residual(Concat(X1,Upsample(X2)))

[0127]

[0128] Where l represents the multi-scale level, Residual (*) represents the residual module, Upsample (*) represents upsampling using bilinear interpolation, and Concat (*) represents the concatenation operation. l ,Y l represents the characteristic size of BEV at scale l, represents the number of BEV feature channels at scale l. In the present invention, l=2,

[0129] Step S8: Perform data preprocessing on the input point cloud and use convolution operation to obtain multi-scale LiDAR BEV features.

[0130] First, the input raw point cloud data is limited in range. The point cloud range used in the present invention is [-54,54]m on the X axis, [-54,54]m on the Y axis, and [-5,3]m on the Z axis. Then the point cloud data is voxelized. The voxel size used in the present invention is [0.075,0.075,0.2]m. Finally, sparse convolution and submanifold sparse convolution are used to extract multi-scale intermediate layer point cloud features. In the present invention, VoxelNet is used to extract features from point clouds, which outputs features of different resolutions during the feature extraction process, indicating that different segmentation granularities are used to encode the location information of the entire space. The intermediate layer point cloud features are compressed to the BEV plane to obtain multi-scale LiDAR BEV features. In the present invention, l=2,

[0131] Step S9: Fusion of multi-scale camera and LiDAR BEV features via deformable attention.

[0132] like Figure 4 As shown, the input multi-scale camera BEV features and multi-scale LiDAR BEV features At all scales Aggregation using deformable attention mechanism as a guide get Finally, different scales are combined Get Features In the present invention (X f ,Y f ,D f )=(180,180,512). The overall process is as follows:

[0133] 1. For scale l, define the input LiDAR BEV feature map and camera BEV feature map:

[0134]

[0135] 2. Use 1*1 convolution π to improve The channel dimension makes it The channel dimensions are the same.

[0136] 3. For the reference point (x l ,y l ), and obtain the feature vectors and The value of The query is added to the positional encoding as a deformable attention mechanism, As value. Deformable attention will produce an offset Δoffset(x l ,y l ) and the weight graph A(x l ,y l ). By calculating (x l ,y l ) and Δoffset(x l ,y l ) can be obtained by adding The location of the sampling point on A(x l ,y l ) to obtain the fusion feature The calculation formula is as follows:

[0137]

[0138] Where M represents the number of attention heads, which is 8 in this invention. N represents the number of sampling points, which is 4 in this invention. m and W′ m Represents a learnable mapping layer.

[0139] 4. Different scales Use bilinear interpolation to downsample the fusion features with higher resolution so that they can be fused with the features with lower resolution. Finally, the fusion features of each scale are combined with Add them together to get F fused :

[0140]

[0141] Step S10: Send the fused features to the LiDAR-based 3D target detection model to obtain a 3D detection frame.

[0142] F fusedAs the input of the 3D object detection model, it performs 3D object recognition. The detection model can adopt any 3D object detection model based on point cloud. The model adopted by the present invention is Transfusion-L. Transfusion-L uses the object detection model DETR to perform 3D object detection. The model initializes a set of object queries as the query of Transformer, and converts the input fusion feature F fused As the key and value of the Transformer, after passing through the Transformer decoder containing multiple attention layers, the center position (x, y, z), size (l, w, h) and yaw angle (yaw) of the 3D object prediction box are obtained.

[0143] The present invention is tested by the public dataset Nuscenes. The Nuscenes dataset is equipped with a variety of sensors and provides rich multimodal data. Nuscenes uses 32-line LiDAR to collect point cloud data, while recording 3D point coordinates and reflection intensity information, providing high-precision data support for the construction of 3D spatial structures and target detection. The dataset contains 6 surround view cameras for collecting high-definition images. In addition, in order to ensure global positioning and data synchronization, Nuscenes is equipped with IMU and GPS.

[0144] Nuscenes covers a wide variety of urban environments, mainly from Boston and Singapore. It contains 1,000 scenes in total, covering a variety of scenarios such as day and night conditions, sunny and rainy weather, and peak and off-peak hours. Nuscenes provides high-quality 3D object detection annotations, with a total of 10 categories involved in the evaluation of 3D detection tasks, using mAP and NDS as indicators for algorithm performance evaluation.

[0145] Table 1 Comparison of full scene detection results of Nuscenes dataset

[0146]

[0147] In order to verify the performance of the method of the present invention, Transfusion is used as the 3D object detector for training, and BEVFusion is used as the baseline. Table 1 shows the verification results of the method of the present invention in the full scene of Nuscenes. As can be seen from Table 1, the accuracy of the method proposed by the present invention is better than that of the previous method.

[0148] In addition, Tables 2 and 3 show the comparison results of detection performance at night and when LiDAR is lost. Compared with the previous method, the method proposed in the present invention can achieve a significant performance improvement in two special scenarios. This shows that the method can improve the robustness of the 3D target detection model, thereby improving the safety of the overall autonomous driving system.

[0149] Table 2 Comparison of nighttime detection results

[0150]

[0151] Table 3 Comparison of LiDAR sensor missing detection results

[0152]

[0153] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program code.

[0154] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. The schemes in the embodiments of the present invention may be implemented in various computer languages, for example, object-oriented programming language Java and literal scripting language JavaScript, etc.

[0155] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0156] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0157] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0158] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0159] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A 3D target detection method for autonomous driving based on multi-sensor fusion, characterized in that: The following steps are involved: Collecting the vehicle posture and the point cloud P and image I of the scene around the vehicle in real time, wherein the point cloud, image and vehicle posture are collected by different sensors; Based on the image I, a dark light enhancement network is used to perform enhancement processing, and the enhanced image Perform image feature extraction to obtain multi-view feature maps Based on the multi-view feature map Performing view transformation to obtain initial camera BEV features, and using the vehicle posture to fuse multiple frames of camera BEV features for updating to obtain final camera BEV features; The final camera BEV feature is sampled to obtain a multi-scale camera BEV feature And extract features from the point cloud to obtain multi-scale LiDAR BEV features The multi-scale camera BEV features and multi-scale LiDAR BEV features Perform feature fusion to obtain fusion feature F fused ; Based on the fusion feature F fused , the 3D target detection model is used to perform three-dimensional target recognition, and the 3D object detection box is output to complete the detection process.

2. The method for 3D target detection based on multi-sensor fusion for autonomous driving according to claim 1, characterized in that: The step of performing enhancement processing on the image I comprises: The image I captured by camera c c Classified as daytime images and night images If it is a daytime image No enhancement processing is performed; If it is a night image An unsupervised dark light enhancement network is used Enhance the image brightness to obtain an enhanced night image The dark light enhancement network is an EnlightenGAN network, and the enhanced night image It is expressed as: Repeat the above steps for the images collected by each camera, and then collect all the daytime images. and all enhanced night images Take the union and get the final enhanced image The enhanced image It is expressed as: In the formula, ∪ represents the union operation.

3. The 3D target detection method for autonomous driving based on multi-sensor fusion according to claim 1, characterized in that: The multi-view feature map is obtained The steps include: Based on the enhanced image An image encoder is used to extract features to obtain a multi-scale feature map, wherein a Swin Transformer is used as a backbone network of the image encoder; A feature pyramid network is used to aggregate the multi-scale feature maps in a cascade manner to obtain a multi-view feature map.

4. The method for 3D target detection based on multi-sensor fusion for autonomous driving according to claim 1, characterized in that: The step of obtaining the initial camera BEV feature comprises: For the multi-view feature map The pixel (u,v) in defines the discrete depth d of the pixel (u,v). i =d0+i*δd,i∈{0,1,…,|N d |}, where d0 is the initial depth, |N d | is the number of discrete depths, δd is the depth interval; For each discrete depth d i , the camera parameter K is used to convert the pixel (u, v) into a point cloud in 3D space, where the conversion expression is: Where (x, y, z) is the point cloud coordinate in 3D space; For the multi-view feature map Perform the above steps for each pixel in to obtain the frustum in 3D space; Based on the multi-view feature map Use the depth estimation network to obtain the depth distribution feature map and contextual features The depth distribution feature map and contextual features Perform outer product on the channel dimension to obtain the weighted feature map The weighted feature map The features in are assigned to the point cloud in the frustum Frustrum to obtain the frustum feature G; The 3D space is voxelized, the point cloud features in each voxel in the cone feature G are aggregated, and pooling is performed along the Z axis to compress the point cloud features to the BEV plane to form an initial camera BEV feature.

5. The method for 3D target detection based on multi-sensor fusion for autonomous driving according to claim 1, characterized in that: The step of obtaining the final camera BEV feature comprises: Based on the vehicle posture, the posture transformation matrix of the vehicle from time t-1 to time t is obtained The pose transformation matrix It is expressed as: In the formula, represents the rotation matrix, represents the translation matrix; Based on the pose transformation matrix The camera BEV feature at time t-1 is transformed by matrix Convert to the BEV coordinate system at the current time t to obtain the BEV features of the camera aligned at time t-1 During the conversion process, the translation vector of the transformation matrix is Multiply by the scale factor between real world and BEV cells; The BEV features of the camera aligned at time t-1 and initial camera BEV characteristics Splice and add position encoding as query, and use the camera BEV features aligned at time t-1 and initial camera BEV characteristics As values, respectively, the deformable attention mechanism is used for calculation to obtain and in and The expression is: Where DeformableAttn(*) is the deformable attention operation, [*] is the channel dimension concatenation, and Pos is the feature position encoding; Using the fully connected layer and The mean of the camera is mapped and compared with the initial camera BEV feature The camera BEV feature at time t is summed and updated as the final camera BEV feature, wherein the final camera BEV feature is expressed as: In the formula, the left side of the equation is is the final camera BEV feature, the right side of the equation is the initial camera BEV feature, MLP(*) is the fully connected layer, and AVERAGE(*) is the averaging operation.

6. The method for 3D target detection based on multi-sensor fusion for autonomous driving according to claim 1, characterized in that: The multi-scale camera BEV features are obtained The steps include: The residual module is used to transform the final camera BEV features Downsampling is performed to obtain features X1 and X2 of different resolutions, where features X1 and X2 are expressed as: X2=Residual(X1) Where, Residual(*) represents the residual module; Feature X2 is upsampled, concatenated with feature X1, and input into the residual module to obtain feature X3, where feature X3 is expressed as: X3=Residual(Concat(X1,Upsample(X2))) In the formula, Upsample(*) represents upsampling using bilinear interpolation, and Concat(*) represents concatenation operation; The final camera BEV features and feature X3 as the multi-scale camera BEV feature The multi-scale camera BEV features It is expressed as: Where l represents the multi-scale level.

7. The method for 3D target detection based on multi-sensor fusion for autonomous driving according to claim 1, characterized in that: The multi-scale LiDAR BEV features are obtained The steps include: Limiting the range of the point cloud on the X-axis, the Y-axis and the Z-axis, and voxelizing the point cloud to obtain a voxelized point cloud; Based on the voxelized point cloud, the VoxelNet network is used to extract point cloud features. During the point cloud feature extraction process, the VoxelNet network compresses the intermediate layer point cloud features to the BEV plane as the multi-scale LiDAR BEV feature.

8. The method for 3D target detection based on multi-sensor fusion for autonomous driving according to claim 7, characterized in that: The obtained fusion feature F fused The steps include: Define multi-scale LiDAR BEV features for scale l and multi-scale camera BEV features Where, X l ,Y l represents the characteristic size of BEV at scale l; Using convolution π to improve The channel dimension of The channel dimensions of are the same; For the reference point (x l ,y l ), and obtain the feature vectors and Will Added to the position encoder as the query of the deformable attention mechanism, As the value of the deformable attention mechanism, the offset Δoffset(x l ,y l ) and weight graph A(x l ,y l ); Based on the offset Δoffset(x l ,y l ), weight graph A(x l ,y l ) and (x l ,y l ) to obtain fusion features of different scales The fusion features of different scales It is expressed as: Where DeformableAttn(*) is the deformable attention, Pos is the feature position code, M is the number of attention heads, N is the number of sampling points, and W is the m and W′ m Represents a learnable mapping layer; Based on the fusion features of different scales The bilinear interpolation method is used for further fusion to obtain the fusion features of each scale and then combined with Add them together to get the final fusion feature F fused , where the final fusion feature F fused It is expressed as: Where Downsample(*) represents downsampling using bilinear interpolation, and L is the number of scales used.

9. The method for 3D target detection based on multi-sensor fusion for autonomous driving according to claim 1, characterized in that: The 3D target detection model is a Transfusion-L model, which uses a DETR method to perform 3D target detection and outputs a 3D object detection frame, wherein the 3D object detection frame includes a center position, a size, and a yaw angle.

10. A computer-readable storage medium, characterized in that: It includes one or more programs for execution by one or more processors of an electronic device, and the one or more programs include instructions for executing a 3D target detection method for autonomous driving with fusion of multiple sensors as described in any one of claims 1-9.

Images