Spatial three-dimensional target detection method and device, equipment and storage medium

Abstract

The invention provides a space three-dimensional target detection method, device and equipment and a storage medium, and aims to unify an infrared image and point cloud data in a BEV space and solve the problem of difference between data structure representation of different sensors and a space coordinate system; point cloud depth supervision and camera parameter priori are introduced into a camera branch to enhance the depth estimation capability of the camera branch, different branches are fused, and fused features and different branches are detected, so that the accuracy of subsequent target detection is improved.

Description

Technical Field

[0001] The present invention belongs to the field of vehicle intelligent driving technology, and in particular relates to a spatial three-dimensional target detection method, device, equipment and storage medium. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Autonomous driving technology has developed rapidly in recent years and is considered to be an inevitable trend in the future automotive industry. Perception plays a key role in autonomous driving. It can accurately perceive the surrounding environment and is the basis for driving intelligence and safety. Object detection is one of the core functions of the perception system. It provides the category and three-dimensional information of the object, which is very important for decision-making and control.

[0004] In related technologies, vision-based 3D target detection only uses images as input. Due to the inadequacy of the depth estimation algorithm, the depth information cannot reach the accuracy of the lidar point cloud. As the lidar detection distance increases, the 3D target detection based on lidar suffers from reduced resolution and poor texture information. Moreover, the current mainstream multimodal detection model based on the image and lidar fusion solution is too complex and has poor scalability. Once a sensor fails, the entire system will not work, making it difficult to deploy in actual application scenarios of autonomous driving. In addition, there are differences in the data structure representation and spatial coordinate system of different sensors. Summary of the invention

[0005] The embodiments of the present application provide a spatial three-dimensional target detection method, device, equipment and storage medium, which can solve the problems of low accuracy of single infrared detection or laser radar detection in related technologies, and the difficulty of deploying and integrating the current multi-modal detection model. The technical solution is as follows:

[0006] In a first aspect, a spatial three-dimensional target detection method is provided, comprising:

[0007] Obtain infrared images and lidar point cloud data of the area to be inspected;

[0008] Calling a three-dimensional object detection model, wherein the three-dimensional object detection model includes at least a camera branch, a radar branch, a fusion branch, and a 3D detection head;

[0009] The camera branch is used to perform feature extraction and fusion operations on the infrared image to obtain multi-scale fusion features; and based on the LSS method, by introducing point cloud depth supervision and camera parameter priors to enhance the depth estimation capability, the obtained multi-scale fusion features are converted into camera BEV features;

[0010] The radar branch is used to extract radar BEV features from the laser radar point cloud data; the fusion branch is used to fuse the camera BEV features and the radar BEV features to obtain fusion features; the 3D detection head is used to perform target detection based on the camera BEV features, the radar BEV features and the fusion features;

[0011] The infrared image and the laser radar point cloud data are input into the three-dimensional target detection model to obtain a target detection result.

[0012] Optionally, the LSS-based method enhances the depth estimation capability by introducing point cloud depth supervision and camera parameter priors, and converts the obtained multi-scale fusion features into camera BEV features, specifically:

[0013] The laser radar point cloud data is converted into a depth map based on the projection matrix, and the depth map is OneHot encoded to obtain the binary depth value of each pixel in the depth map;

[0014] The camera parameters are processed by a fully connected layer, and the result of the processing is multiplied by the multi-scale fusion feature to obtain a multi-scale fusion image feature containing the camera parameters;

[0015] The binary depth value of each pixel in the depth map is used as supervision information, and the multi-scale fusion image features including camera parameters are supervised based on the depth loss function to obtain pseudo voxel features.

[0016] Optionally, the obtained pseudo voxel features are encoded as BEV space features, and the camera BEV features are extracted from the BEV space features through convolution, normalization, and ReLU function operations in sequence.

[0017] Optionally, the fusion branch is used to fuse the camera BEV feature and the radar BEV feature to obtain a fusion feature, specifically:

[0018] Fusing the camera BEV feature and the radar BEV feature to obtain a preliminary fused feature;

[0019] Performing a convolution operation on the result of the preliminary fusion through a first convolution layer, and performing an operation on the result of the first convolution layer operation through global average pooling to obtain a channel-level global feature;

[0020] The global features of the channel level are operated sequentially through the second convolution layer and the Sigmoid activation function to obtain a gating value;

[0021] The obtained gated value is multiplied by the result after the first convolutional layer operation to obtain the final fusion feature.

[0022] Optionally, the loss function of the 3D detection head is the sum of category loss, bounding box loss and depth loss.

[0023] Optionally, the laser radar point cloud data is augmented to obtain an augmented matrix; and the transformation matrix from the camera coordinate system to the radar coordinate system is updated according to the augmented matrix to ensure spatial consistency between the BEV features generated in the camera branch and the BEV features generated in the radar branch.

[0024] Optionally, the radar branch is used to convert the lidar point cloud data into a Pillar form to obtain a Pillar feature; and expand the obtained Pillar feature to obtain a pseudo image; and perform feature extraction on the pseudo image to obtain a radar BEV feature.

[0025] In a second aspect, a spatial three-dimensional object detection device is provided, comprising:

[0026] An acquisition module is used to acquire infrared images and laser radar point cloud data of the area to be detected;

[0027] A calling module, used to call a three-dimensional target detection model, wherein the three-dimensional target detection model includes at least a camera branch, a radar branch, a fusion branch and a 3D detection head;

[0028] The camera branch is used to perform feature extraction and fusion operations on the infrared image to obtain multi-scale fusion features; and based on the LSS method, by introducing point cloud depth supervision and camera parameter priors to enhance the depth estimation capability, the obtained multi-scale fusion features are converted into camera BEV features;

[0029] The radar branch is used to extract radar BEV features from the laser radar point cloud data; the fusion branch is used to fuse the camera BEV features and the radar BEV features to obtain fusion features; the 3D detection head is used to perform target detection based on the camera BEV features, the radar BEV features and the fusion features;

[0030] A processing module is used to input the infrared image and the laser radar point cloud data into the three-dimensional target detection model to obtain a target detection result.

[0031] A third aspect provides an electronic device, including:

[0032] processor;

[0033] a memory for storing processor-executable instructions;

[0034] Wherein, the processor is configured to implement a spatial three-dimensional target detection method described in the first aspect above.

[0035] According to a fourth aspect, a computer-readable storage medium is provided, on which instructions are stored. When the instructions are executed by a processor, a spatial three-dimensional target detection method described in the first aspect is implemented.

[0036] One or more of the above technical solutions have the following beneficial effects:

[0037] In the present invention, the laser radar and infrared camera are set as independent branches for easy deployment; the BEV space, i.e., the bird's-eye view space, is selected as the same representation of two different modalities, and the camera branch and the radar branch respectively unify the infrared image in the two-dimensional space and the point cloud data in the three-dimensional space into the BEV space, solving the problem of data structure representation of different sensors and the difference in spatial coordinate systems, which is more friendly to subsequent multimodal feature fusion and three-dimensional target detection tasks. Point cloud depth supervision and camera parameter priors are introduced into the camera branch to enhance the depth estimation capability of the camera branch, and different branches are fused, and the fused features are detected with the output results of different branches to improve the accuracy of subsequent target detection.

[0038] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0040] Figure 1 is a flowchart of a spatial three-dimensional object detection method according to an exemplary embodiment;

[0041] Figure 2 is a structural diagram of a design model according to an exemplary embodiment;

[0042] Figure 3 is a structural diagram of an image encoding module according to an exemplary embodiment;

[0043] Figure 4 is a diagram of an improved deep network structure in an image-BEV view transformation module according to an exemplary embodiment;

[0044] Figure 5 is a BEV pooling acceleration kernel diagram in an image-to-BEV view transformation module according to an exemplary embodiment;

[0045] Figure 6 is a radar branch network structure diagram according to an exemplary embodiment;

[0046] Figure 7 is a structural diagram of a gated attention fusion mechanism module in a fusion module according to an exemplary embodiment;

[0047] FIG. 8( a ) is a diagram showing a data acquisition hardware platform and its layout according to an exemplary embodiment. Figure 1 ;

[0048] FIG8( b ) is a diagram showing a data acquisition hardware platform and its layout according to an exemplary embodiment. Figure 2 ;

[0049] Fig. 9 is a visualization result of a challenging scenario deployment shown according to an exemplary embodiment;

[0050] Fig.10 is a schematic diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0051] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0052] It should be noted that the terms used herein are for describing specific embodiments only and are not intended to be limiting of exemplary embodiments according to the present invention.

[0053] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0054] Before introducing in detail the spatial three-dimensional object detection method provided in the embodiment of the present application, a brief introduction to the application scenarios and implementation environment involved in the embodiment of the present application is first given.

[0055] First, the application scenarios involved in the embodiments of the present application are briefly introduced.

[0056] Infrared sensors and LiDARs are important sensors. Infrared images contain texture information, and laser point clouds provide spatial geometric information. Their fusion can improve the vehicle's perception of the real world and achieve a higher level of autonomous driving. In the case of complex traffic road conditions, multi-sensor fusion is the best solution to achieve high-level autonomous driving, and will become the core driving force for achieving advanced autonomous driving. However, there are differences in the data structure representation and spatial coordinate system of different sensors, which makes the adaptability of target detection poor. To this end, an embodiment of the present application provides a spatial three-dimensional target detection method, which can adaptively fuse the data collected by infrared sensors and LiDARs. For its specific implementation, please refer to the following embodiments.

[0057] Next, the implementation environment involved in the embodiments of the present application is briefly introduced.

[0058] The spatial three-dimensional target detection method based on infrared and laser radar fusion provided in the embodiment of the present application can be executed by an electronic device, which can be a terminal, or the electronic device can also be an embedded device. Further, the terminal can be a laptop computer, a desktop computer, a portable computer, a tablet computer, etc., which is not limited in the embodiment of the present application.

[0059] This embodiment discloses a spatial three-dimensional target detection method based on infrared and laser radar fusion, including:

[0060] Step 101: Acquire infrared images and laser radar point cloud data of the area to be detected;

[0061] Step 201: Call the 3D object detection model, such as Figure 2 As shown, the 3D target detection model includes at least a camera branch, a radar branch, a fusion branch and a 3D detection head;

[0062] The camera branch is used to extract and fuse infrared images to obtain multi-scale fusion features. Based on the LSS method, the depth estimation capability is enhanced by introducing point cloud depth supervision and camera parameter priors, and the obtained multi-scale fusion features are converted into camera BEV features.

[0063] The radar branch is used to extract radar BEV features from the lidar point cloud data; the fusion branch is used to fuse the camera BEV features and the radar BEV features to obtain fused features; the 3D detection head is used to detect targets based on the camera BEV features, radar BEV features and fused features;

[0064] Step 301: Input the infrared image and the lidar point cloud data into a three-dimensional target detection model to obtain a target detection result.

[0065] In this embodiment, the camera branch and the radar branch respectively unify the infrared image in the two-dimensional space and the point cloud data in the three-dimensional space into the BEV space, which solves the problems of data structure representation and spatial coordinate system differences of different sensors; point cloud depth supervision and camera parameter priors are introduced into the camera branch to enhance the depth estimation capability of the camera branch, and different branches are fused, and the fused features are detected with the output results of different branches to improve the accuracy of subsequent target detection.

[0066] Optionally, based on the LSS method, the depth estimation capability is enhanced by introducing point cloud depth supervision and camera parameter priors, and the obtained multi-scale fusion features are converted into camera BEV features, specifically:

[0067] The laser radar point cloud data is converted into a depth map based on the projection matrix, and the depth map is OneHot encoded to obtain the binary depth value of each pixel in the depth map;

[0068] The camera parameters are processed by a fully connected layer, and the result of the processing is multiplied by the multi-scale fusion feature to obtain a multi-scale fusion image feature containing the camera parameters;

[0069] The binary depth value of each pixel in the depth map is used as supervision information, and the multi-scale fusion image features containing camera parameters are supervised based on the depth loss function to obtain pseudo voxel features.

[0070] Optionally, the obtained pseudo voxel features are encoded as BEV space features, and the camera BEV features are extracted from the BEV space features through convolution, normalization, and ReLU function operations in sequence.

[0071] Optionally, the fusion branch is used to fuse the camera BEV feature and the radar BEV feature to obtain a fusion feature, specifically:

[0072] The camera BEV features and the radar BEV features are fused to obtain preliminary fusion features;

[0073] The result of the preliminary fusion is convolved through the first convolutional layer, and the result of the first convolutional layer operation is processed through global average pooling to obtain channel-level global features;

[0074] The global features at the channel level are operated through the second convolutional layer and the Sigmoid activation function to obtain the gate value;

[0075] The obtained gated value is multiplied by the result after the first convolutional layer operation to obtain the final fusion feature.

[0076] Optionally, the loss function of the 3D detection head is the sum of category loss, bounding box loss and depth loss.

[0077] Optionally, data augmentation is performed on the lidar point cloud data to obtain an augmented matrix; and a transformation matrix from the camera coordinate system to the radar coordinate system is updated according to the augmented matrix to ensure spatial consistency between the BEV features generated in the camera branch and the BEV features generated in the radar branch.

[0078] Optionally, the radar branch is used to convert the lidar point cloud data into a Pillar form to obtain a Pillar feature; and to expand the obtained Pillar feature to obtain a pseudo image; and to perform feature extraction on the pseudo image to obtain a radar BEV feature.

[0079] Figure 1According to another exemplary flowchart of a spatial three-dimensional target detection method based on infrared and laser radar fusion, this embodiment is illustrated by applying the spatial three-dimensional target detection method based on infrared and laser radar fusion to electronic equipment. The spatial three-dimensional target detection method based on infrared and laser radar fusion may include the following steps:

[0080] Step 101: Obtain infrared images and lidar point cloud data of the area to be detected.

[0081] In this embodiment, a data acquisition hardware platform is formed by a MEMS laser radar and an infrared camera. The data acquisition hardware platform and its layout are as follows: Figure 8(a)-Figure 8(b) As shown, the LiDAR point cloud data and infrared image are acquired by the MEMS LiDAR and the infrared camera respectively.

[0082] Step 201: calling a three-dimensional target detection model, the three-dimensional target detection model at least includes a camera branch, a radar branch, a fusion branch and a 3D detection head; the camera branch is used to extract features and fuse infrared images to obtain multi-scale fusion features; and based on the LSS method, by introducing point cloud depth supervision and camera parameter priors to enhance the depth estimation capability, the obtained multi-scale fusion features are converted into camera BEV features; the radar branch is used to extract radar BEV features from lidar point cloud data; the fusion branch is used to fuse camera BEV features and radar BEV features to obtain fusion features; the 3D detection head is used to perform target detection based on camera BEV features, radar BEV features and fusion features.

[0083] Before calling the 3D object detection model, multiple training samples can be obtained, and the 3D object detection model to be trained is trained based on the obtained training samples to obtain a trained 3D object detection model. Exemplarily, the multiple training samples include a data set consisting of 16095 radar point clouds and 16095 infrared images with a resolution of 640×512, and the data set is annotated. The data set includes three categories: car, pedestrian, and cyclist, covering various scenes such as city, countryside, and night.

[0084] The training samples are preprocessed and the class-balanced sampling CBGS strategy is used for training to solve the class imbalance problem.

[0085] The three-dimensional object detection model proposed in this embodiment includes a camera branch, a radar branch, a fusion branch, and a 3D detection head. The camera branch structure includes an image encoding module, an image-BEV view transformation module, and a BEV encoding module connected in sequence.

[0086] The image encoding module mainly consists of three parts: the backbone network, a neck network and a multi-scale feature fusion (MSFF) module. The backbone network and the neck network output multi-scale features H / 8×W / 8×C, H / 16×W / 16×C, and H / 32×W / 32×C. Then, the multi-scale feature fusion (MSFF) module is used to perform adaptive average pooling on H / 8×W / 8×C, and the feature map size remains unchanged. H / 16×W / 16×C and H / 32×W / 32×C are upsampled to H / 8×W / 8×C respectively. Then, the pooled features and the upsampled features are fused, and the multi-scale fusion features with a shape of H / 8×W / 8×3C are obtained through the convolution layer; where C is the number of channels, W is the width of the image, and H is the height of the image.

[0087] In order to efficiently and accurately extract the multi-scale features of the input image, taking into account both accuracy and speed, and having good robustness and generalization ability, the following Figure 3 The network structure shown completes feature extraction and obtains the feature map

[0088] Specifically, Figure 3 As shown, the backbone network includes a Focus layer, a first convolution block, a first CSP module, a second convolution block, a second CSP module, a third convolution block, a third CSP module, a fourth convolution block, an SPPF module and a fourth CSP module, which are connected in sequence.

[0089] The Neck network includes a fifth convolution block, a first upsampling, a first concat layer, a fifth CSP module, a sixth convolution block, a second upsampling, a second concat layer, and a sixth CSP module connected in sequence; and a first downsampling layer, a third concat layer, a seventh CSP module, a second downsampling layer, a fourth concat layer, and an eighth CSP module connected in sequence to the sixth CSP module; wherein the fifth convolution block is connected to the fourth concat layer; the third CSP module is connected to the first concat layer; the second CSP module is connected to the second concat layer; and the sixth convolution block is connected to the third concat layer.

[0090] The MSFF module includes three branches. The first branch is adaptAvgpool and the seventh convolution block connected in sequence; the second branch is the third upsampling and the eighth convolution block connected in sequence; the third branch is the fourth upsampling and the ninth convolution block connected in sequence; among them, the sixth CSP module is connected to the first branch; the seventh CSP module is connected to the second branch; and the eighth CSP module is connected to the third branch.

[0091] Among them, the Focus layer includes four parallel slice operations, and the results of the four slice operations are sequentially operated through the concat layer and the Conv layer.

[0092] like Figure 4 As shown in the figure, the image-BEV view transformation module uses the LSS method to transform image features Transformed into camera BEV features. Based on the deep network of the Lift stage in LSS, the improved deep network introduces a camera parameter prior module and a deep supervision module to enhance the depth estimation capability of the camera branch.

[0093] The camera parameter prior module uses the camera parameters corresponding to the acquired infrared image as the prior for depth estimation. The camera parameters are processed by the fully connected layer to upgrade the dimension to the number of image feature channels, and then combined with the image features. Multiply them together to get the image feature D containing the camera parameters pred .

[0094] The depth supervision module uses the real point cloud for supervision. First, the acquired LiDAR point cloud is converted into a depth map DepthMap through the projection matrix, and then the depth map DepthMap is encoded using OneHot to obtain the binary depth D of each pixel. gt , the binary depth value of each pixel in the depth map is used as supervision information, and the multi-scale fusion image features containing camera parameters are supervised based on the depth loss function to obtain pseudo voxel features. For the depth loss, the binary cross entropy loss is used to constrain the deep network. The calculation of the depth loss is:

[0095]

[0096] Among them, S represents the Sigmoid activation function, S(-D pred )=1-S(D pred ).

[0097] Improved deep network structure such as Figure 4 As shown in the figure, the frustum point cloud features (H×W×D pred ×C) is projected into the corresponding cell, and then the BEV pooling operation is performed to obtain the pseudo voxel feature F p-voxel ∈R C×X×Y×Z .

[0098] The BEV encoding module pools the pseudo-voxel features F obtained by the above BEV p-voxel ∈R C×X×Y×Z Encoded as BEV spatial feature F BEV ∈R X×Y×(ZC), the Z-dimensional spatial information is retained, and then a simple CBR (Conv+Batchnorm+Relu) network is used to extract the camera BEV features to maximize the retention of spatial information and reduce the loss cost.

[0099] This embodiment also designs a BEV pooling acceleration kernel, builds a GPUs parallel computing engine based on the CUDA platform, allocates a GPU thread to each BEV grid, and designs a GPU kernel function to realize parallel acceleration of cumulative summation calculation, reducing the model training time from 72 hours to 30 hours. Figure 5 As shown, after BEV pooling, the pseudo voxel feature F is obtained. p-voxel ∈R C×X×Y×Z .

[0100] The radar branch adapts to any SOTA 3D point cloud detection model. In this embodiment, Figure 6 As shown, in the radar branch, the original point cloud is processed using the point cloud columnarization method to obtain the Pillar feature F pillar ∈R P×C , P represents the number of Pillars, C represents the number of channels, expand P to (W, H), and get the pseudo image F p-img ∈R W×H×C , radar BEV features are obtained through the Second backbone network and Neck network.

[0101] In order to align the two features in the same BEV space, the point cloud data is augmented and its augmented matrix is ​​saved. Then, the transformation matrix from the camera coordinate system to the radar coordinate system is updated to ensure the spatial consistency of the BEV features generated in the camera branch and the BEV features generated in the radar branch.

[0102] Since the laser radar and infrared camera are installed at different positions of the vehicle and are based on their own coordinate systems, it is necessary to calculate the rigid body transformation matrix between the two coordinate systems through multi-sensor joint registration. The rigid body transformation matrix is ​​shown in the following formula:

[0103]

[0104] Among them, R is a 3×3 rotation matrix, T is a 3×1 translation matrix, [X L ,Y L ,Z L ] represents the coordinates in the radar coordinate system, [X C ,Y C ,Z C ] represents the coordinates in the camera coordinate system. The function of the rotation matrix R is to unify the basis vectors of the radar coordinate system and the camera coordinate system. The function of the translation matrix T is to translate the origins of the two coordinate systems to a unified position.

[0105] The two sensors can use the multi-sensor registration algorithm to solve the accurate rotation matrix R and translation matrix T, which can realize the mutual transformation between the radar coordinate system and the camera coordinate system.

[0106] Among them, in terms of data augmentation, global rotation, scaling, translation, horizontal flipping and other data augmentation are used for point clouds, and cropping, scaling and horizontal flipping and other data augmentation are used for images.

[0107] The fusion branch unifies the two-dimensional space and three-dimensional space of the camera branch and radar branch into the BEV space respectively, and uses the gated attention fusion mechanism module to connect the BEV features from different branches through the convolution layer to obtain the fused features. Then, the global average pooling is used to obtain the channel-level global features. A convolution layer and Sigmoid activation function are added to learn the weights of each channel as the gating value to indicate the importance of each channel. Finally, the fused features are multiplied by the gated value channel by channel to output the fused features with enhanced channel attention.

[0108] like Figure 7 As shown in the figure, the gated attention fusion mechanism module includes a concat layer, a first convolution layer, an adaptiveAvgpool, a second convolution layer, and a Sigmoid activation function connected in sequence; the output of the first convolution layer is multiplied by the result of the Sigmoid activation function.

[0109] 3D detection head. The camera branch, radar branch and fusion branch of this embodiment all use the CenterPoint based on Anchor-Free as the 3D detection head. For the category loss L class Using the Focal loss function, for the 3D bounding box loss L bbox Using Smooth L1 loss function, for the depth loss L depth A binary cross entropy loss function is used. The final loss is the sum of category loss, bounding box loss, and depth loss, as shown below:

[0110]

[0111] Among them, y∈{0,1} represents the true value, S represents the Sigmoid activation function, α and γ are hyperparameters, and S(-x)=1-S(x).

[0112]

[0113] Among them, y represents the true value and f(x) represents the predicted value.

[0114] Loss = λ 1 L class +λ 2 L bbox+λ 3 L depth

[0115] Among them, λ 1 , 2 , 3 Represents the weight value of each loss.

[0116] This embodiment selects BEV space as the unified representation of two different modalities. The camera branch and radar branch unify the two-dimensional space and three-dimensional space into BEV space respectively, solving the problem of data structure representation and spatial coordinate system differences of different sensors, which is more friendly to subsequent multimodal feature fusion and three-dimensional target detection tasks. The deep network of the camera branch is improved, and the point cloud depth supervision module and camera parameter prior module are introduced to enhance the depth estimation capability of the camera branch. A GPU acceleration kernel is designed in the image-BEV view transformation to improve the model detection speed, and the running rate on the GPU reaches 33FPS. The bird's-eye view spatial three-dimensional target detection method based on infrared and lidar fusion in this embodiment has been successfully deployed on the MIIVII APEX AD10 embedded AI computing platform.

[0117] The comparison of the performance indicators of the solution in this embodiment with the performance indicators of the existing solution is shown in Table 1.

[0118] In this experiment, the detection ranges of the X-axis, Y-axis, and Z-axis are set to [0m, +120m], [-33.6m, +33.6m], and [-2m, +4m], respectively. The training set consists of 16,095 images and the validation set consists of 3,219 images. During the training process, the data is divided into batchsize = 8 groups, and the Adam gradient descent method is used for 60 rounds of training. The initial learning rate is set to 0.00001 until the loss function converges and the network training is completed. The model is deployed to AD10, which receives data streams from the two sensors in real time and performs real-time inference, and is published to RVIZ for visualization.

[0119] Table 1: Performance index test results

[0120]

[0121] Fig. 9 Deployment visualization results for the challenging scenarios shown.

[0122] According to an exemplary embodiment, a spatial three-dimensional object detection device can be implemented by software, hardware, or a combination of the two. The spatial three-dimensional object detection device may include:

[0123] An acquisition module is used to acquire infrared images and laser radar point cloud data of the area to be detected;

[0124] A calling module, used to call a 3D target detection model, which includes at least a camera branch, a radar branch, a fusion branch and a 3D detection head;

[0125] The camera branch is used to extract and fuse infrared images to obtain multi-scale fusion features. Based on the LSS method, the depth estimation capability is enhanced by introducing point cloud depth supervision and camera parameter priors, and the obtained multi-scale fusion features are converted into camera BEV features.

[0126] The radar branch is used to extract radar BEV features from the lidar point cloud data; the fusion branch is used to fuse the camera BEV features and the radar BEV features to obtain fused features; the 3D detection head is used to detect targets based on the camera BEV features, radar BEV features and fused features;

[0127] The processing module is used to input the infrared image and lidar point cloud data into the three-dimensional target detection model to obtain the target detection result.

[0128] Optionally, the processing module is used to: based on the LSS method, enhance the depth estimation capability by introducing point cloud depth supervision and camera parameter priors, and convert the obtained multi-scale fusion features into camera BEV features, specifically:

[0129] The laser radar point cloud data is converted into a depth map based on the projection matrix, and the depth map is OneHot encoded to obtain the binary depth value of each pixel in the depth map;

[0130] The camera parameters are processed by a fully connected layer, and the result of the processing is multiplied by the multi-scale fusion feature to obtain a multi-scale fusion image feature containing the camera parameters;

[0131] The binary depth value of each pixel in the depth map is used as supervision information, and the multi-scale fusion image features containing camera parameters are supervised based on the depth loss function to obtain pseudo voxel features.

[0132] Optionally, the processing module is used to: encode the obtained pseudo voxel features into BEV space features, and extract camera BEV features from the BEV space features through convolution, normalization, and ReLU function operations in sequence.

[0133] Optionally, the processing module is used for: the fusion branch is used for fusing the camera BEV feature and the radar BEV feature to obtain a fusion feature, specifically:

[0134] The camera BEV features and the radar BEV features are fused to obtain preliminary fusion features;

[0135] The result of the preliminary fusion is convolved through the first convolutional layer, and the result of the first convolutional layer operation is processed through global average pooling to obtain channel-level global features;

[0136] The global features at the channel level are operated through the second convolutional layer and the Sigmoid activation function to obtain the gate value;

[0137] The obtained gated value is multiplied by the result after the first convolutional layer operation to obtain the final fusion feature.

[0138] Optionally, the processing module is used to: the loss function of the 3D detection head is the sum of category loss, bounding box loss and depth loss.

[0139] Optionally, the processing module is used to: perform data augmentation on the lidar point cloud data to obtain an augmented matrix; update the transformation matrix from the camera coordinate system to the radar coordinate system according to the augmented matrix to ensure spatial consistency between the BEV features generated in the camera branch and the BEV features generated in the radar branch.

[0140] Optionally, the processing module is used for: the radar branch is used to convert the lidar point cloud data into a Pillar form to obtain a Pillar feature; and the obtained Pillar feature is expanded to obtain a pseudo image; and feature extraction is performed on the pseudo image to obtain a radar BEV feature.

[0141] In an embodiment of the present application, the camera branch and the radar branch respectively unify the infrared image in the two-dimensional space and the point cloud data in the three-dimensional space into the BEV space, thereby solving the problems of differences in data structure representation and spatial coordinate systems of different sensors; point cloud depth supervision and camera parameter priors are introduced into the camera branch to enhance the depth estimation capability of the camera branch, and different branches are fused, and the fused features are detected with the output results of different branches to improve the accuracy of subsequent target detection.

[0142] It should be noted that: the spatial three-dimensional target detection device provided in the above embodiment only uses the division of the above-mentioned functional modules as an example to illustrate when implementing the spatial three-dimensional target detection method. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the spatial three-dimensional target detection device provided in the above embodiment and the spatial three-dimensional target detection method embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0143] A block diagram of an electronic device provided by an exemplary embodiment of the present application. The electronic device may be a smart phone, a tablet computer, an MP3 player (Moving Picture Group Audio Layer III), an MP4 player (Moving Picture Group Audio Layer IV), a laptop computer or a desktop computer. The electronic device may also be referred to as a user device, a portable terminal, a laptop terminal, a desktop terminal or other names.

[0144] Typically, an electronic device includes a processor and a memory.

[0145] The processor may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor may be implemented in at least one of the following hardware forms: DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor may also include a main processor and a coprocessor. The main processor is a processor used to process data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor used to process data in the standby state.

[0146] In some embodiments, the processor may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor may also include an AI (artificial intelligence) processor, which is used to process computing operations related to machine learning.

[0147] The memory may include one or more computer-readable storage media, which may be non-transitory. The memory may also include high-speed random access memory, and non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory is used to store at least one instruction, which is used to be executed by the processor to implement a spatial three-dimensional target detection method provided in the method embodiment of the present application.

[0148] In some embodiments, the electronic device may optionally include: a peripheral device interface and at least one peripheral device. The processor, the memory and the peripheral device interface may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit, a touch display screen, a camera, an audio circuit, a positioning component and a power supply.

[0149] The peripheral device interface can be used to connect at least one peripheral device related to I / O (Input / Output) to the processor and the memory. In some embodiments, the processor, the memory, and the peripheral device interface are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor, the memory, and the peripheral device interface can be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0150] The radio frequency circuit is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit communicates with communication networks and other communication devices through electromagnetic signals. The radio frequency circuit converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the radio frequency circuit includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The radio frequency circuit can communicate with other devices through at least one wireless communication protocol. The wireless communication protocol includes but is not limited to: the World Wide Web, a metropolitan area network, an intranet, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit may also include circuits related to NFC (Near Field Communication), which is not limited in this application.

[0151] The display screen is used to display the UI (User Interface). The UI may include graphics, text, icons, videos and any combination thereof. When the display screen is a touch display screen, the display screen also has the ability to collect touch signals on the surface or above the surface of the display screen. The touch signal can be input to the processor as a control signal for processing. At this time, the display screen can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen can be one, and the front panel of the electronic device is set; in other embodiments, the display screen can be at least two, which are respectively set on different surfaces of the electronic device or are folded; in some other embodiments, the display screen can be a flexible display screen, which is set on the curved surface or folded surface of the electronic device. Even, the display screen can also be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen can be made of materials such as LCD (liquid crystal display) and OLED (Organic Light-Emitting Diode).

[0152] The camera assembly is used to capture images or videos. Optionally, the camera assembly includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the electronic device, and the rear camera is arranged on the back of the electronic device. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize the panoramic shooting and VR (virtual reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0153] The audio circuit may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into a processor for processing, or input them into a radio frequency circuit to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively arranged in different parts of the electronic device. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert electrical signals from the processor or radio frequency circuit into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert electrical signals into sound waves audible to humans, but also convert electrical signals into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit may also include a headphone jack.

[0154] The positioning component is used to locate the current geographic location of the electronic device to achieve navigation or LBS (Location Based Service). The positioning component can be a positioning component based on the US GPS (Global Positioning System), China's Beidou system or a foreign Galileo system.

[0155] The power source is used to power various components in the electronic device. The power source can be AC, DC, a disposable battery, or a rechargeable battery. When the power source includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged through a wired line, and a wireless rechargeable battery is a battery that is charged through a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0156] In some embodiments, the electronic device further includes one or more sensors, including but not limited to: an acceleration sensor, a gyroscope sensor, a pressure sensor, a fingerprint sensor, an optical sensor, and a proximity sensor.

[0157] The accelerometer can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established by the electronic device. For example, the accelerometer can be used to detect the components of gravity acceleration on the three coordinate axes. The processor can control the touch screen to display the user interface in a horizontal view or a vertical view according to the gravity acceleration signal collected by the accelerometer. The accelerometer can also be used to collect game or user motion data.

[0158] The gyroscope sensor can detect the body direction and rotation angle of the electronic device. The gyroscope sensor can work with the acceleration sensor to collect the user's 3D actions on the electronic device. The processor can implement the following functions based on the data collected by the gyroscope sensor: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

[0159] The pressure sensor can be set on the side frame of the electronic device and / or the lower layer of the touch display screen. When the pressure sensor is set on the side frame of the electronic device, it can detect the user's grip signal of the electronic device, and the processor performs left and right hand recognition or quick operation according to the grip signal collected by the pressure sensor. When the pressure sensor is set on the lower layer of the touch display screen, the processor controls the operability controls on the interface according to the user's pressure operation on the touch display screen. The operability controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

[0160] The fingerprint sensor is used to collect the user's fingerprint, and the processor identifies the user's identity based on the fingerprint collected by the fingerprint sensor, or the fingerprint sensor identifies the user's identity based on the collected fingerprint. When the user's identity is identified as a trusted identity, the processor authorizes the user to perform relevant sensitive operations, which include unlocking the screen, viewing encrypted information, downloading software, paying, and changing settings. The fingerprint sensor can be set on the front, back or side of the electronic device. When a physical button or manufacturer logo is set on the electronic device, the fingerprint sensor can be integrated with the physical button or manufacturer logo.

[0161] The optical sensor is used to collect the ambient light intensity. In one embodiment, the processor can control the display brightness of the touch display screen according to the ambient light intensity collected by the optical sensor. Specifically, when the ambient light intensity is high, the display brightness of the touch display screen is increased; when the ambient light intensity is low, the display brightness of the touch display screen is decreased. In another embodiment, the processor can also dynamically adjust the shooting parameters of the camera assembly according to the ambient light intensity collected by the optical sensor.

[0162] A proximity sensor, also known as a distance sensor, is usually provided on the front panel of an electronic device. The proximity sensor is used to collect the distance between the user and the front of the electronic device. In one embodiment, when the proximity sensor detects that the distance between the user and the front of the electronic device is gradually decreasing, the processor controls the touch display screen to switch from the screen-on state to the screen-off state; when the proximity sensor detects that the distance between the user and the front of the electronic device is gradually increasing, the processor controls the touch display screen to switch from the screen-off state to the screen-on state.

[0163] Those skilled in the art will understand that Fig.10 The structure shown in the figure does not constitute a limitation on the electronic device, and may include more or less components than shown in the figure, or combine certain components, or adopt a different component arrangement.

[0164] An embodiment of the present application also provides a non-temporary computer-readable storage medium. When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute a spatial three-dimensional target detection method provided by the above-mentioned embodiments.

[0165] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0166] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A spatial three-dimensional target detection method, characterized in that: include: Obtain infrared images and lidar point cloud data of the area to be inspected; Calling a three-dimensional object detection model, wherein the three-dimensional object detection model includes at least a camera branch, a radar branch, a fusion branch, and a 3D detection head; The camera branch is used to perform feature extraction and fusion operations on the infrared image to obtain multi-scale fusion features; Based on the LSS method, the depth estimation capability is enhanced by introducing point cloud depth supervision and camera parameter priors, and the obtained multi-scale fusion features are converted into camera BEV features. The radar branch is used to extract radar BEV features from the laser radar point cloud data; the fusion branch is used to fuse the camera BEV features and the radar BEV features to obtain fusion features; the 3D detection head is used to perform target detection based on the camera BEV features, the radar BEV features and the fusion features; The infrared image and the laser radar point cloud data are input into the three-dimensional target detection model to obtain a target detection result.

2. A spatial three-dimensional target detection method as claimed in claim 1, characterized in that: The LSS-based method introduces point cloud depth supervision and camera parameter priors to enhance depth estimation capabilities, and converts the obtained multi-scale fusion features into camera BEV features, specifically: The laser radar point cloud data is converted into a depth map based on the projection matrix, and the depth map is OneHot encoded to obtain the binary depth value of each pixel in the depth map; The camera parameters are processed by a fully connected layer, and the result of the processing is multiplied by the multi-scale fusion feature to obtain a multi-scale fusion image feature containing the camera parameters; The binary depth value of each pixel in the depth map is used as supervision information, and the multi-scale fusion image features including camera parameters are supervised based on the depth loss function to obtain pseudo voxel features.

3. A spatial three-dimensional target detection method as claimed in claim 2, characterized in that: The obtained pseudo voxel features are encoded into BEV space features, and the camera BEV features are extracted from the BEV space features through convolution, normalization, and ReLU function operations in sequence.

4. A spatial three-dimensional target detection method as claimed in claim 1, characterized in that: The fusion branch is used to fuse the camera BEV feature and the radar BEV feature to obtain a fusion feature, specifically: Fusing the camera BEV feature and the radar BEV feature to obtain a preliminary fused feature; Performing a convolution operation on the result of the preliminary fusion through a first convolution layer, and performing an operation on the result of the first convolution layer operation through global average pooling to obtain a channel-level global feature; The global features of the channel level are operated sequentially through the second convolution layer and the Sigmoid activation function to obtain a gating value; The obtained gated value is multiplied by the result after the first convolutional layer operation to obtain the final fusion feature.

5. A spatial three-dimensional target detection method as claimed in claim 1 or 4, characterized in that: The laser radar point cloud data is augmented to obtain an augmented matrix; the conversion matrix from the camera coordinate system to the radar coordinate system is updated according to the augmented matrix to ensure spatial consistency between the BEV features generated in the camera branch and the BEV features generated in the radar branch.

6. A spatial three-dimensional target detection method as claimed in claim 1, characterized in that: The radar branch is used to convert the laser radar point cloud data into Pillar form to obtain Pillar features; and expand the obtained Pillar features to obtain a pseudo image; and perform feature extraction on the pseudo image to obtain radar BEV features.

7. A spatial three-dimensional target detection method as claimed in claim 1, characterized in that: The loss function of the 3D detection head is the sum of category loss, bounding box loss and depth loss.

8. A spatial three-dimensional target detection device, characterized in that: include: An acquisition module is used to acquire infrared images and laser radar point cloud data of the area to be detected; A calling module, used to call a three-dimensional target detection model, wherein the three-dimensional target detection model includes at least a camera branch, a radar branch, a fusion branch and a 3D detection head; The camera branch is used to perform feature extraction and fusion operations on the infrared image to obtain multi-scale fusion features; Based on the LSS method, the depth estimation capability is enhanced by introducing point cloud depth supervision and camera parameter priors, and the obtained multi-scale fusion features are converted into camera BEV features. The radar branch is used to extract radar BEV features from the laser radar point cloud data; the fusion branch is used to fuse the camera BEV features and the radar BEV features to obtain fusion features; the 3D detection head is used to perform target detection based on the camera BEV features, the radar BEV features and the fusion features; A processing module is used to input the infrared image and the laser radar point cloud data into the three-dimensional target detection model to obtain a target detection result.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to implement the steps of any one of the methods described in claims 1-7.

10. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instructions are executed by a processor, the steps of any method described in claims 1-7 are implemented.

Images