3D target detection method and device for roadside system and electronic equipment

Abstract

The application discloses a 3D target detection method and device for a roadside system and electronic equipment, wherein the method comprises the following steps: acquiring a mapping relationship between a camera image and a laser radar point cloud of a fixed machine position in a fixed field of view through joint calibration; performing feature extraction on the camera image through a 2D target detection network to generate a first feature map; splicing a feature vector of a voxel of the laser radar point cloud with a feature vector at a corresponding position in the first feature map to generate a splicing matrix; performing feature extraction on a representation vector in the splicing matrix through a multi-layer VFE to generate a second feature map; inputting the second feature map into a 3D target detection network for detection; and outputting a 3D detection result through the 3D target detection network. The technical scheme of the application reduces the data volume and improves the operation speed through feature layer fusion under the premise of avoiding the loss of key information as much as possible.

Description

Technical Field

[0001] This application relates to a 3D target detection method and apparatus, and more particularly to a 3D target detection method, apparatus and electronic equipment for roadside systems. Background Technology

[0002] Target detection technology in roadside environmental perception systems acquires information about the surrounding environment through roadside sensors, primarily including static and dynamic target information in traffic scenarios. Common sensors used for perception include cameras, LiDAR, and millimeter-wave radar, which are mainly mounted on bridges or traffic light structures. Roadside environmental perception is susceptible to various factors such as weather and the limitations of the sensors themselves, and it is generally impossible to achieve perception of the entire traffic scenario using a single sensor. Therefore, multi-sensor fusion is required to jointly complete the traffic information perception task. However, multi-sensor fusion itself also has many shortcomings and drawbacks.

[0003] Existing image and point cloud-based fusion perception methods are generally divided into two categories based on the fusion level: pre-fusion and post-fusion. Pre-fusion typically includes data-layer fusion and feature-layer fusion. Data-layer fusion does not require data preprocessing and directly fuses the data, while feature-layer fusion first obtains the features of the image or point cloud and then performs feature fusion processing. For example, a 3D point cloud is first projected onto an image to obtain bird's-eye view features and front view features. Then, the bird's-eye view features, front view features, and image features are input into a hierarchical fusion network. Finally, the fused features are used in an object classification network and 3D bounding boxes for prediction. Data-layer fusion, because it directly processes the original data, requires high bandwidth and computing power. Furthermore, without preprocessing the original data, the fusion result is susceptible to interference from erroneous information and has poor robustness. While feature-layer fusion reduces the amount of data and improves the computation speed, the discarding of some data may lead to the loss of hidden key information. Summary of the Invention

[0004] According to embodiments of this application, a target detection method, apparatus, and electronic device for a roadside system are provided.

[0005] In a first aspect of this application, a target detection method for a roadside system is provided, comprising: The mapping relationship between camera images and lidar point clouds at fixed positions within a fixed field of view is obtained through joint calibration. The mapping relationship is the mapping relationship between the pixel coordinate system of the camera image and the lidar coordinate system of the lidar point cloud. A first feature map is generated by extracting features from the camera image using a 2D object detection network. The lidar point cloud is voxelized to form multiple voxels. According to the mapping relationship, the lidar point cloud is projected onto the camera image, so that multiple voxels in the lidar point cloud form a one-to-one correspondence with multiple positions in the first feature map. The feature vector of the voxel is concatenated with the feature vector of the position in the first feature map that has the one-to-one correspondence with the voxel to generate a concatenation matrix; A second feature map is generated by extracting features from the representation vectors in the concatenated matrix using a multi-layer VFE. The second feature map is input into a 3D object detection network for detection, and the 3D object detection network outputs the 3D detection result.

[0006] In some embodiments, the step of generating a first feature map by extracting features from the camera image using a 2D object detection network includes: The 2D object detection network was pre-trained using the COCO dataset; The pre-trained 2D object detection network was fine-tuned using an object detection dataset corresponding to a specific roadside traffic scene; The first feature map is generated by extracting features from the camera image using the finely tuned 2D object detection network.

[0007] In some embodiments, the 2D object detection network used is Faster. The RCNN structure, where the first feature map is a Faster... The result of feature map layer extraction in RCNN structure.

[0008] In some embodiments, the voxelization process of the lidar point cloud to form multiple voxels includes: The lidar point cloud is divided into three-dimensional spaces. Let the depth, height, and width of the lidar point cloud be (D, H, W). Multiple voxels are formed by dividing the lidar point cloud using cubes of the same size. Each voxel has a depth, width, and height of (VD, VH, VW). The number of voxels... ; Each point in the lidar point cloud is assigned to a corresponding voxel, and the average coordinate of all points within the voxel is denoted as (V). x V y V z If ), then the vector representation of each point after voxelization is p. i =(x i ,y i ,z i ,r i ,x i V x ,y i V y ,z i V z ), where x, y, z are the coordinates of the point in three-dimensional space, and r is the reflectivity of the lidar; T points are randomly sampled from each non-empty voxel, and points less than T are padded with 0.

[0009] In some embodiments, the step of concatenating the feature vector of the voxel with the feature vector of the position in the first feature map that has the one-to-one correspondence with the voxel to generate a concatenation matrix includes: The first feature map is reduced in dimensionality using a two-layer FCN, which uses ReLU as the activation function and batch normalization. The dimensionality change of the first feature map is as follows: 512 >96 >16; The feature vector of the voxel is concatenated with the feature vector of the voxel in the first feature map after dimensionality reduction, which has a one-to-one correspondence with the voxel, to generate a concatenation matrix. The concatenation matrix is ​​denoted as [N,T,C], where N is the number of non-empty voxels, T is the number of sampling points in the voxel, and C is the dimension of the concatenation matrix, C = 7 + 16, where 7 is the vector dimension of the voxel feature and 16 is the final dimension of the first feature map after dimensionality reduction.

[0010] In some embodiments, the step of using multi-layer VFE to extract features from the representation vectors in the concatenated matrix to generate a second feature map includes: The spliced ​​matrix is ​​then increased in dimensionality using the same FCN layer; Max pooling is applied to the feature vectors of multiple point clouds in the voxel features of the up-dimensional splicing matrix to generate a single feature vector. The first feature vector is copied to obtain multiple point cloud copies from the voxel features, and then concatenated with the feature vectors of the multiple point clouds to obtain the second feature map.

[0011] In some embodiments, the step of inputting the second feature map into a 3D object detection network for detection and outputting a 3D detection result through the 3D object detection network includes: The second feature map input is downsampled three times using three sets of 2D convolutions, and the feature map is downsampled to 1 / 2, 1 / 4 and 1 / 8 of the input size in sequence to form a 1 / 2 downsampled feature map, a 1 / 4 downsampled feature map and a 1 / 8 downsampled feature map; The transposed convolution is used to expand the three downsampled feature maps of different sizes by 2 times in the depth dimension, and the 1 / 4 downsampled feature map and the 1 / 8 downsampled feature map are upsampled to 1 / 2 times the original input size. The 1 / 2 downsampled feature map is augmented by a factor of 2 in depth, and the 1 / 4 downsampled feature map and the 1 / 8 downsampled feature map are augmented by a factor of 2 in depth and then upsampled to half the original input size. These are then concatenated and input into the prediction part. The prediction part uses two detection heads to output confidence scores and 3D regression boxes, respectively. Each detection head is composed of a 2D convolution head of a different size.

[0012] In a second aspect of this application, a 3D target detection device for a roadside system is provided, comprising: The mapping relationship acquisition module is used to acquire the mapping relationship between camera images and lidar point clouds at fixed positions within a fixed field of view through joint calibration. The mapping relationship is the mapping relationship between the pixel coordinate system of the camera image and the radar coordinate system of the lidar point cloud. The first feature map generation module is used to extract features from the camera image through a 2D object detection network to generate a first feature map. The voxelization module performs voxelization processing on the lidar point cloud to form multiple voxels; The projection module is used to project the lidar point cloud onto the camera image according to the mapping relationship. This establishes a one-to-one correspondence between multiple voxels in the lidar point cloud and multiple locations in the first feature map; The splicing matrix generation module is used to splice the feature vector of the voxel with the feature vector of the position in the first feature map that has the one-to-one correspondence with the voxel to generate a splicing matrix; The second feature map generation module is used to extract features from the representation vectors in the concatenated matrix using a multi-layer VFE to generate a second feature map. The 3D target detection module is used to input the second feature map into the 3D target detection network for detection, and output the 3D detection result through the 3D target detection network.

[0013] In some embodiments, the first feature map generation module includes: The pre-training submodule is used to pre-train the 2D object detection network using the COCO dataset; The fine-tuning submodule is used to fine-tune the pre-trained 2D object detection network using the object detection dataset corresponding to the determined roadside traffic scene; The feature extraction submodule is used to extract features from the camera image using the fine-tuned 2D object detection network.

[0014] In a third aspect of this application, an electronic device is provided, including at least one processor and a memory, the memory storing a program and configured to have at least one processor perform the following steps: The mapping relationship between camera images and lidar point clouds at fixed positions within a fixed field of view is obtained through joint calibration. The mapping relationship is the mapping relationship between the pixel coordinate system of the camera image and the lidar coordinate system of the lidar point cloud. A first feature map is generated by extracting features from the camera image using a 2D object detection network. The lidar point cloud is voxelized to form multiple voxels. According to the mapping relationship, the lidar point cloud is projected onto the camera image, so that multiple voxels in the lidar point cloud form a one-to-one correspondence with multiple positions in the first feature map. The feature vector of the voxel is concatenated with the feature vector of the position in the first feature map that has the one-to-one correspondence with the voxel to generate a concatenation matrix; A second feature map is generated by extracting features from the representation vectors in the concatenated matrix using a multi-layer VFE. The second feature map is input into a 3D object detection network for detection, and the 3D object detection network outputs the 3D detection result.

[0015] The beneficial technical effect of this application's technical solution lies in obtaining the mapping relationship between camera images and LiDAR point clouds at fixed positions within a fixed field of view through joint calibration. Based on the mapping relationship, the LiDAR point cloud is projected onto the camera image, so that multiple voxels in the LiDAR point cloud form a one-to-one correspondence with multiple positions in the first feature map. The feature vectors of the voxels and the feature vectors of the positions in the first feature map that have the one-to-one correspondence with the voxels are used to generate a stitching matrix through feature fusion. Furthermore, the detection of 3D targets is achieved based on the second feature map extracted from the stitching matrix. While minimizing the loss of key information, this significantly reduces the amount of data and improves the detection speed of 3D targets in the roadside system. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the method flow structure of an embodiment of this application; Figure 2 This is a flowchart illustrating a specific method according to an embodiment of this application; Figure 3 yes Figure 2 Step 220 in the method flowchart is a further method flowchart; Figure 4 It's Faster RCNN architecture diagram; Figure 5 yes Figure 2 Step 230 in the method flowchart is a further method flowchart; Figure 6 This is a schematic diagram of point cloud voxelization; Figure 7 This is a VFE structure diagram; Figure 8 It is a diagram of a two-layer VFE structure; Figure 9 This is a block diagram of the device structure according to an embodiment of this application; Figure 10 yes Figure 7 The submodule structure diagram of module 720 in the embodiment. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to specific embodiments and accompanying drawings. It should be understood that these descriptions are merely exemplary and not intended to limit the scope of this application. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0018] Obviously, the described embodiments are only a part of the embodiments of this application, not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0019] In the description of this application, it should be noted that the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, the technical features involved in the different embodiments of this application described below may be combined with each other as long as they do not conflict with each other.

[0020] Technical terms: The COCO (Common Objects in Context) dataset is a large and rich dataset for object detection, segmentation, and captioning.

[0021] PointNet: A deep network for processing point cloud data, used to process LiDAR point cloud data.

[0022] VoxelNet: A hierarchical learning network based on point cloud-based 3D spatial information.

[0023] RPN (Region Proposal Network): A neural network used to generate candidate regions (Region Proposals).

[0024] 3D Convolutional Middle Layers: 3D convolution.

[0025] 3D RPN: Includes 3D convolution and RPN, a neural network used to generate 3D candidate regions.

[0026] RCNN: An algorithm model that introduces CNN (Convolutional Neural Network) into the field of object detection. It uses CNN to extract features of candidate regions. The specific steps include: ① Identify approximately 1000 in the image. 2000 candidate boxes (using selective search); ② The image patches within each candidate box are scaled to the same size and then input into the CNN for feature extraction; ③ Use a classifier to determine whether the features extracted from the candidate boxes belong to a specific class; ④ For candidate boxes belonging to a certain feature, use a regressor to further adjust their positions.

[0027] Fast RCNN: An improved algorithm model of RCNN, the specific steps of which include: ① Identify approximately 1000 in the image. 2000 candidate boxes (using selective search); ② Input the entire image into the CNN to obtain the feature map; ③ Find the mapping patch of each candidate box on the feature map, and use this patch as the convolutional feature of each candidate box as input to the SPP layer and subsequent layers; ④ Use a classifier to determine whether the features extracted from the candidate boxes belong to a specific class; ⑤ For candidate boxes belonging to a certain feature, use a regressor to further adjust their positions.

[0028] Faster RCNN: A further improved algorithm model of RCNN, the specific steps of which include: ① Input the entire image into a CNN to obtain a feature map; ② The convolutional features are input into the RPN to obtain the feature information of the candidate boxes; ③ Use a classifier to determine whether the features extracted from the candidate boxes belong to a specific class; ④ For candidate boxes belonging to a certain feature, use a regressor to further adjust their positions.

[0029] Batch normalization: a normalization method for intermediate layers of a network.

[0030] Max pooling: A type of pooling or downsampling method.

[0031] ReLU: A type of activation function.

[0032] The technical solution of this application mainly focuses on improving the feature layer fusion technology solution in the prior art. For example... Figure 1 As shown, to fuse information from camera images and LiDAR point cloud data, reduce data volume, improve processing speed, and prevent the loss of key information hidden in the camera images, this application's technical solution addresses the fixed field of view characteristic of roadside systems. First, joint calibration is used to obtain the mapping relationship between camera images and LiDAR point clouds. Then, a pre-trained 2D target detection network is used to encode features from the camera images, primarily using the last convolutional layer of the 2D detection network to extract features, ensuring that the output feature map encodes semantic information from the camera images. This serves as prior knowledge to assist the roadside system in target judgment in subsequent steps. Next, using the calibration results, points or voxels from the LiDAR point cloud are projected onto the camera image, and the corresponding pixel features in the feature map are concatenated with the point or voxel features in the LiDAR point cloud as input to a subsequent 3D target detection network to predict the 3D detection results.

[0033] This application proposes a multi-mode 3D target detection fusion scheme that combines camera images and LiDAR point clouds. By enhancing point cloud features with image semantic features and fusing image and point cloud features in the early learning stage, the loss of key information hidden in the image or point cloud is avoided, ultimately achieving more accurate and comprehensive 3D target detection.

[0034] Figure 2 This is a flowchart illustrating a specific method in an embodiment of this application.

[0035] like Figure 2 As shown, a 3D target detection method for roadside systems includes: Step 210: Obtain the mapping relationship between the camera image and the lidar point cloud at a fixed position within a fixed field of view through joint calibration. The mapping relationship is the mapping relationship between the pixel coordinate system of the camera image and the radar coordinate system of the lidar point cloud.

[0036] Step 220: Extract features from the camera image using a 2D object detection network to generate a first feature map.

[0037] Step 230: Perform voxelization on the lidar point cloud to form multiple voxels.

[0038] Step 240: Based on the mapping relationship, project the lidar point cloud onto the camera image, so that multiple voxels in the lidar point cloud form a one-to-one correspondence with multiple positions in the first feature map.

[0039] Step 250: Concatenate the feature vector of the voxel with the feature vector of the position in the first feature map that has the one-to-one correspondence with the voxel to generate a concatenation matrix.

[0040] Step 260: Use multi-layer VFE to extract features from the representation vectors in the spliced ​​matrix to generate a second feature map.

[0041] Step 270: Input the second feature map into the 3D target detection network for detection, and output the 3D detection result through the 3D target detection network.

[0042] In some embodiments, step 220 involves using a 2D object detection network to extract features from the camera image to generate a first feature map, as detailed in [reference needed]. Figure 3 Further method flowchart.

[0043] like Figure 3 As shown, the first feature map is generated by extracting features from the camera image using a 2D object detection network, including: Step 221: Pre-train the 2D object detection network using the COCO dataset; Step 222: Fine-tune the pre-trained 2D object detection network using the object detection dataset corresponding to the determined roadside traffic scene; Step 223: Extract features from the camera image using the finely tuned 2D object detection network to generate a first feature map.

[0044] Figure 4 It's Faster A schematic diagram of the RCNN structure. (See diagram below.) Figure 4 As shown, in steps 221 to 223 above, a first feature map is generated by extracting features from the camera image using a 2D object detection network. The detection network used is Faster. The RCNN architecture involves pre-training on the COCO dataset, followed by fine-tuning on a roadside traffic scene object detection dataset. During network learning, the RPN (Region Proposal Network) and RCNN modules share the underlying feature maps as input, enabling the first feature map to semantically encode the input camera image. Finally, Faster... The feature map layer in RCNN is used as the result of extracting the first feature map.

[0045] In some embodiments, step 230 involves voxelizing the lidar point cloud to form multiple voxels, as can be seen in [reference needed]. Figure 5 Further method flowchart.

[0046] like Figure 5 As shown, the lidar point cloud is voxelized to form multiple voxels, including: Step 231: Divide the lidar point cloud into three-dimensional space. Let the depth, height, and width of the lidar point cloud be (D, H, W). Divide the lidar point cloud into multiple voxels using cubes of the same size. Each voxel has a depth, width, and height of (V). D V H V W The number of voxels ; Step 232: Assign each point in the lidar point cloud to the corresponding voxel, and denote the average value of the coordinates of all points within the voxel as , then the vector representation of each point after voxelization is , where x, y, z are the coordinates of the point in three-dimensional space, and r is the reflectivity of the lidar. Step 233: Randomly sample T points from each non-empty voxel, padding with 0 if the number of points is less than T.

[0047] Figure 6 This is a voxelized diagram of a lidar point cloud. (Example:) Figure 6 As shown, steps 231 to 233 above involve voxelizing the lidar point cloud to form multiple voxels, specifically including: (1) Voxel Partition: The LiDAR point cloud is partitioned in three-dimensional space. Assuming the depth, height, and width of the input point cloud are (D, H, W), it is partitioned using cubes of the same size. Each voxel has a depth, width, and height of (V). D V H V W If the input point cloud is given, then the number of Voxels generated is 1. The model uses VD=0.4, VH=0.2, and VW=0.2.

[0048] (2) Vector Representation (Grouping): Each point in the lidar point cloud is assigned to a corresponding voxel, and the average coordinate of all points within a voxel is denoted as (V0). x V y V z The vector representation of the lidar point cloud after voxelization is p. i =(x i ,y i ,zi ,r i ,x i V x ,y i V y ,z i V z x, y, z are coordinates in space, and r is the reflectivity of the lidar.

[0049] (3) Random Sampling: In order to represent each non-empty voxel, T points are randomly sampled from each non-empty voxel. If the number of points is less than T, zeros are added. This can effectively reduce memory usage and computation.

[0050] In some embodiments, step 250, which involves concatenating the feature vector of the voxel with the feature vectors of the positions in the first feature map that have the one-to-one correspondence with the voxel to generate a concatenation matrix, includes: The first feature map is reduced in dimensionality using a two-layer FCN, which uses ReLU as the activation function and batch normalization. The dimensionality change of the first feature map is as follows: 512 >96 >16; The feature vector of the voxel is concatenated with the feature vector of the voxel in the first feature map after dimensionality reduction, which has a one-to-one correspondence with the voxel, to generate a concatenation matrix. The concatenation matrix is ​​denoted as [N,T,C], where N is the number of non-empty voxels, T is the number of sampling points in the voxel, and C is the dimension of the concatenation matrix, C = 7 + 16, where 7 is the vector dimension of the voxel feature and 16 is the final dimension of the first feature map after dimensionality reduction.

[0051] Since each feature point in the first feature map obtained from the feature extraction part has a dimension of 512, using this dimension as input for the next stage would increase the computational load. Therefore, before concatenating the vectors with the LiDAR point cloud, the feature points of the image can be dimensionality reduced. In this embodiment, a two-layer fully connected network is used. Dimensionality reduction is performed on a connected network (FCN). FCN uses ReLU as the activation function and batch normalization. The final dimensionality change is: 512. >96 >16. Based on the mapping relationship between the LiDAR point cloud and the camera image, link the point cloud and image features to achieve the fusion of image features into the point cloud. The final input dimension is denoted as: [N,T,C], where N is the number of non-empty voxels, T is the number of sampling points in the voxels, and C is the dimension of the concatenated point cloud and image features, where C is 7+16.

[0052] In some embodiments, step 260, which involves using a multi-layer VFE to extract features from the representation vectors in the concatenated matrix to generate a second feature map, includes: The spliced ​​matrix is ​​then increased in dimensionality using the same FCN layer; Max pooling is applied to the feature vectors of multiple point clouds in the voxel features of the up-dimensional splicing matrix to generate a single feature vector. The first feature vector is copied to obtain multiple point cloud copies from the voxel features, and then concatenated with the feature vectors of the multiple point clouds to obtain the second feature map.

[0053] Figure 7 This is a VFE structure diagram. (Example) Figure 7 As shown, in the above implementation, a fully connected layer (FCN) can first be used to encode each concatenated point cloud vector according to the vector dimension to obtain point-by-point vector features. The VFE structure first performs max pooling on each pointwise vector feature according to the element's dimension to obtain locally aggregated features. Finally, it concatenates the locally aggregated feature vectors to the pointwise vector features to obtain the feature representation of each VFE structure for the input. The fully connected layer FCN, for each input Point... Wise vectors share weights, and the fully connected layer includes Batch Normal and ReLU.

[0054] Figure 8 It is a two-layer VFE structure diagram, as shown below. Figure 8 As shown, this embodiment further utilizes a two-layer VEF structure to represent the network input, and then uses max pooling to pool all representation vectors in the voxel along the element dimension, ultimately obtaining a single vector to represent the voxel. The specific dimensions used in this embodiment are as follows: VFE 1. Output dimensions: (7+16, 32) VFE 2. Output dimensions: (32, 128) Max pooling: Pooling is performed element by element, and the output feature dimension is [N, 1, 128], which is equivalent to representing each voxel as a vector of (1, 128).

[0055] In some embodiments, step 270, which involves inputting the second feature map into a 3D object detection network for detection and outputting a 3D detection result through the 3D object detection network, includes: The second feature map input is downsampled three times using three sets of 2D convolutions, and the feature map is downsampled to 1 / 2, 1 / 4 and 1 / 8 of the input size in sequence to form a 1 / 2 downsampled feature map, a 1 / 4 downsampled feature map and a 1 / 8 downsampled feature map; The transposed convolution is used to expand the three downsampled feature maps of different sizes by 2 times in the depth dimension, and the 1 / 4 downsampled feature map and the 1 / 8 downsampled feature map are upsampled to 1 / 2 times the original input size. The 1 / 2 downsampled feature map is augmented by a factor of 2 in depth, and the 1 / 4 downsampled feature map and the 1 / 8 downsampled feature map are augmented by a factor of 2 in depth and then upsampled to half the original input size. These are then concatenated and input into the prediction part. The prediction part uses two detection heads to output confidence scores and 3D regression boxes, respectively. Each detection head is composed of a 2D convolution head of a different size.

[0056] In the above steps, the extracted second feature map is input into the 3D RPN structure for final prediction. The 3DRPN includes 3D Convolutional Middle Layers and RPN.

[0057] After obtaining the feature representation of the entire point cloud using a two-layer VEF structure, each non-empty voxel is first remapped back to the original 3D space. Considering the feature vectors encoded by the voxels, the encoded features are finally combined into a sparse 4D tensor, such as [N, 1, 128]. >[128,D',H',W'], and then use multi-layer 3D convolution to further abstract the features of this 4D feature representation. Finally, keep the size of the H' and W' dimensions to convert the 4D features into 3D features. The converted features can be understood as a representation on the ground plane, and it is assumed that the objects do not have a height dimension stacking on the 3D drop.

[0058] Based on the description of the above embodiments, taking the input size in the embodiment of step 260 as an example, the entire network processing feature size change process is as follows:

[0059] After feature extraction, the extracted second feature map is input into a 3D RPN structure for final prediction. The 3D RPN detector first uses three sets of 2D convolutions to downsample the input feature map three times, successively downsampling it to 1 / 2, 1 / 4, and 1 / 8 of the input size. Then, transposed convolutions are used to double the depth dimension of the three downsampled feature maps, and the 1 / 4 and 1 / 8 downsampled feature maps are upsampled to 1 / 2 of their original input and output sizes. Finally, the three feature maps are concatenated for the final prediction. The prediction part uses two detection heads, which output confidence scores and 3D regression boxes, respectively. Each convolution head is composed of 2D convolution heads of different sizes.

[0060] The final output format is: (1) Confidence scale: [W,D,anchors] (2) 3D regression box size: [W,D,7*anchors], the 7 dimensions are (x,y,z,l,w,h,Θ), representing the center point, length, width, height and heading angle.

[0061] Here, anchors represents the number of anchor boxes preset for each feature point on the final feature layer of the 3D detection network. The length, width, and height of the anchors are preset directly based on the size of the target object type in the real world.

[0062] Figure 9 This is a block diagram of the device structure according to an embodiment of this application.

[0063] like Figure 9 As shown, a 3D target detection device for a roadside system includes: The mapping relationship acquisition module 710 is used to acquire the mapping relationship between the camera image and the lidar point cloud at a fixed position within a fixed field of view through joint calibration. The mapping relationship is the mapping relationship between the pixel coordinate system of the camera image and the lidar coordinate system of the lidar point cloud. The first feature map generation module 720 is used to extract features from the camera image through a 2D object detection network to generate a first feature map. The voxelization processing module 730 performs voxelization processing on the lidar point cloud to form multiple voxels; The projection module 740 is used to project the lidar point cloud onto the camera image according to the mapping relationship, so that multiple voxels in the lidar point cloud form a one-to-one correspondence with multiple positions in the first feature map. The splicing matrix generation module 750 is used to splice the feature vector of the voxel with the feature vector of the position in the first feature map that has the one-to-one correspondence with the voxel to generate a splicing matrix; The second feature map generation module 760 is used to extract features from the representation vectors in the concatenated matrix using a multi-layer VFE to generate a second feature map. The 3D target detection module 770 is used to input the second feature map into the 3D target detection network for detection, and output the 3D detection result through the 3D target detection network.

[0064] Figure 10 yes Figure 9 A block diagram of the submodules of module 720 in the embodiment.

[0065] In some embodiments, the first feature map generation module includes: The pre-training submodule 721 is used to pre-train the 2D object detection network using the COCO dataset; Fine-tuning submodule 722 is used to fine-tune the pre-trained 2D object detection network using an object detection dataset corresponding to a determined roadside traffic scene; The feature extraction submodule 723 is used to extract features from the camera image through the fine-tuned 2D object detection network.

[0066] This application also provides an electronic device, including at least one processor and a memory, wherein the memory stores a program and is configured to have at least one processor execute the following steps: The mapping relationship between camera images and lidar point clouds at fixed positions within a fixed field of view is obtained through joint calibration. The mapping relationship is the mapping relationship between the pixel coordinate system of the camera image and the lidar coordinate system of the lidar point cloud. A first feature map is generated by extracting features from the camera image using a 2D object detection network. The lidar point cloud is voxelized to form multiple voxels. According to the mapping relationship, the lidar point cloud is projected onto the camera image, so that multiple voxels in the lidar point cloud form a one-to-one correspondence with multiple positions in the first feature map. The feature vector of the voxel is concatenated with the feature vector of the position in the first feature map that has the one-to-one correspondence with the voxel to generate a concatenation matrix; A second feature map is generated by extracting features from the representation vectors in the concatenated matrix using a multi-layer VFE. The second feature map is input into a 3D object detection network for detection, and the 3D object detection network outputs the 3D detection result.

[0067] The above embodiments illustrate devices, modules, or units, which can be implemented by computer chips or physical entities, or by products with certain functions. For ease of description, the above devices are described by functional modules. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware.

[0068] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs) containing computer-usable program code. The form of a computer program product implemented on ROM, optical memory, etc.

[0069] This application can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0070] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A 3D target detection method for roadside systems, comprising: The mapping relationship between camera images and lidar point clouds at fixed positions within a fixed field of view is obtained through joint calibration. The mapping relationship is the mapping relationship between the pixel coordinate system of the camera image and the lidar coordinate system of the lidar point cloud. A first feature map is generated by extracting features from the camera image using a 2D object detection network. The lidar point cloud is voxelized to form multiple voxels. According to the mapping relationship, the lidar point cloud is projected onto the camera image, so that multiple voxels in the lidar point cloud form a one-to-one correspondence with multiple positions in the first feature map. The feature vector of the voxel is concatenated with the feature vector of the position in the first feature map that has the one-to-one correspondence with the voxel to generate a concatenation matrix; A second feature map is generated by extracting features from the representation vectors in the concatenated matrix using a multi-layer VFE. The second feature map is input into a 3D object detection network for detection, and the 3D object detection network outputs the 3D detection result. The step of concatenating the feature vector of the voxel with the feature vector of the position in the first feature map that has the one-to-one correspondence with the voxel to generate a concatenation matrix includes: The first feature map is reduced in dimensionality using two layers of FCN. The FCN uses ReLU as the activation function and batch norm for normalization. The dimensionality change process of the first feature map is: 512->96->16. The feature vector of the voxel is concatenated with the feature vector of the voxel in the first feature map after dimensionality reduction, which has a one-to-one correspondence with the voxel, to generate a concatenation matrix. The concatenation matrix is ​​denoted as [N,T,C], where N is the number of non-empty voxels, T is the number of sampling points in the voxel, and C is the dimension of the concatenation matrix, C = 7 + 16, where 7 is the vector dimension of the voxel feature and 16 is the final dimension of the first feature map after dimensionality reduction.

2. The 3D object detection method according to claim 1, wherein the feature extraction of the camera image through a 2D object detection network to generate a first feature map includes: The 2D object detection network was pre-trained using the COCO dataset; The pre-trained 2D object detection network was fine-tuned using an object detection dataset corresponding to a specific roadside traffic scene; The first feature map is generated by extracting features from the camera image using the finely tuned 2D object detection network.

3. The 3D object detection method according to claim 1 or 2, wherein the 2D object detection network adopts a Faster-RCNN structure, and the first feature map is the extraction result of the feature map layer in the Faster-RCNN structure.

4. The 3D target detection method according to claim 1, wherein the voxelization process of the lidar point cloud to form multiple voxels includes: The lidar point cloud is divided into three-dimensional spaces. Let the depth, height, and width of the lidar point cloud be (D, H, W). Multiple voxels are formed by dividing the lidar point cloud using cubes of the same size. Then, the depth, width, and height of each voxel are (V...). D V H V W The number of voxels Each point in the lidar point cloud is assigned to a corresponding voxel, and the average coordinate of all points within the voxel is denoted as (V). x V y V z If ), then the vector representation of each point after voxelization is p. i =(x i ,y i ,z i ,r i ,x i -V x ,y i -V y ,z i -V z ), where x, y, z are the coordinates of the point in three-dimensional space, and r is the reflectivity of the lidar; T points are randomly sampled from each non-empty voxel, and points less than T are padded with 0.

5. The target detection method according to claim 1, wherein the method of using multi-layer VFE to extract features from the representation vectors in the concatenated matrix to generate a second feature map includes: The spliced ​​matrix is ​​then increased in dimensionality using the same FCN layer; Max pooling is applied to the feature vectors of multiple point clouds in the voxel features of the up-dimensional splicing matrix to generate a single feature vector. The first feature vector is copied to obtain multiple point cloud copies from the voxel features, and then concatenated with the feature vectors of the multiple point clouds to obtain the second feature map.

6. The target detection method according to claim 1, wherein inputting the second feature map into a 3D target detection network for detection and outputting a 3D detection result through the 3D target detection network includes: The second feature map input is downsampled three times using three sets of 2D convolutions, and the feature map is downsampled to 1 / 2, 1 / 4 and 1 / 8 of the input size in sequence to form a 1 / 2 downsampled feature map, a 1 / 4 downsampled feature map and a 1 / 8 downsampled feature map; The transposed convolution is used to expand the three downsampled feature maps of different sizes by 2 times in the depth dimension, and the 1 / 4 downsampled feature map and the 1 / 8 downsampled feature map are upsampled to 1 / 2 times the original input size. The 1 / 2 downsampled feature map is augmented by a factor of 2 in depth, and the 1 / 4 downsampled feature map and the 1 / 8 downsampled feature map are augmented by a factor of 2 in depth and then upsampled to half the original input size. These are then concatenated and input into the prediction part. The prediction part uses two detection heads to output confidence scores and 3D regression boxes, respectively. Each detection head is composed of a 2D convolution head of a different size.

7. A 3D target detection device for a roadside system, comprising: The mapping relationship acquisition module is used to acquire the mapping relationship between camera images and lidar point clouds at fixed positions within a fixed field of view through joint calibration. The mapping relationship is the mapping relationship between the pixel coordinate system of the camera image and the radar coordinate system of the lidar point cloud. The first feature map generation module is used to extract features from the camera image through a 2D object detection network to generate a first feature map. The voxelization module performs voxelization processing on the lidar point cloud to form multiple voxels; The projection module is used to project the lidar point cloud onto the camera image according to the mapping relationship, so that multiple voxels in the lidar point cloud form a one-to-one correspondence with multiple positions in the first feature map. The splicing matrix generation module is used to splice the feature vector of the voxel with the feature vector of the position in the first feature map that has the one-to-one correspondence with the voxel to generate a splicing matrix; The second feature map generation module is used to extract features from the representation vectors in the concatenated matrix using a multi-layer VFE to generate a second feature map. The 3D target detection module is used to input the second feature map into the 3D target detection network for detection, and output the 3D detection result through the 3D target detection network; The step of concatenating the feature vector of the voxel with the feature vector of the position in the first feature map that has the one-to-one correspondence with the voxel to generate a concatenation matrix includes: The first feature map is reduced in dimensionality using two layers of FCN. The FCN uses ReLU as the activation function and batch norm for normalization. The dimensionality change process of the first feature map is: 512->96->16. The feature vector of the voxel is concatenated with the feature vector of the voxel in the first feature map after dimensionality reduction, which has a one-to-one correspondence with the voxel, to generate a concatenation matrix. The concatenation matrix is ​​denoted as [N,T,C], where N is the number of non-empty voxels, T is the number of sampling points in the voxel, and C is the dimension of the concatenation matrix, C = 7 + 16, where 7 is the vector dimension of the voxel feature and 16 is the final dimension of the first feature map after dimensionality reduction.

8. The 3D target detection device according to claim 7, wherein the first feature map generation module comprises: The pre-training submodule is used to pre-train the 2D object detection network using the COCO dataset; The fine-tuning submodule is used to fine-tune the pre-trained 2D object detection network using the object detection dataset corresponding to the determined roadside traffic scene; The feature extraction submodule is used to extract features from the camera image using the fine-tuned 2D object detection network.

9. An electronic device comprising at least one processor and a memory, the memory storing a program and configured to have at least one processor perform the following steps: The mapping relationship between camera images and lidar point clouds at fixed positions within a fixed field of view is obtained through joint calibration. The mapping relationship is the mapping relationship between the pixel coordinate system of the camera image and the lidar coordinate system of the lidar point cloud. A first feature map is generated by extracting features from the camera image using a 2D object detection network. The lidar point cloud is voxelized to form multiple voxels. According to the mapping relationship, the lidar point cloud is projected onto the camera image, so that multiple voxels in the lidar point cloud form a one-to-one correspondence with multiple positions in the first feature map. The feature vector of the voxel is concatenated with the feature vector of the position in the first feature map that has the one-to-one correspondence with the voxel to generate a concatenation matrix; A second feature map is generated by extracting features from the representation vectors in the concatenated matrix using a multi-layer VFE. The second feature map is input into a 3D object detection network for detection, and the 3D object detection network outputs the 3D detection result. The step of concatenating the feature vector of the voxel with the feature vector of the position in the first feature map that has the one-to-one correspondence with the voxel to generate a concatenation matrix includes: The first feature map is reduced in dimensionality using two layers of FCN. The FCN uses ReLU as the activation function and batch norm for normalization. The dimensionality change process of the first feature map is: 512->96->16. The feature vector of the voxel is concatenated with the feature vector of the voxel in the first feature map after dimensionality reduction, which has a one-to-one correspondence with the voxel, to generate a concatenation matrix. The concatenation matrix is ​​denoted as [N,T,C], where N is the number of non-empty voxels, T is the number of sampling points in the voxel, and C is the dimension of the concatenation matrix, C = 7 + 16, where 7 is the vector dimension of the voxel feature and 16 is the final dimension of the first feature map after dimensionality reduction.

Images