A processing method and apparatus for point cloud target detection based on fused features.

Abstract

This invention relates to a processing method and apparatus for point cloud target detection based on fused features. The method includes: acquiring a first point cloud; extracting features from the first point cloud to generate a forward-looking feature tensor and fusing the features with the first point cloud to generate a first point cloud tensor; extracting point cloud column features from the first point cloud tensor to generate point cloud column features and extracting bird's-eye view features from the point cloud column features to generate bird's-eye view features; extracting deformable convolutional features from the bird's-eye view features and fusing the convolutional features with the bird's-eye view features; and performing target detection based on the fused bird's-eye view features. This invention enhances the feature richness of point clouds and improves target detection accuracy by performing multi-view, multi-scale feature extraction and fusion of point clouds.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a processing method and device for point cloud target detection based on fused features. BACKGROUND

[0002] The perception module of an autonomous driving system often uses point cloud-based 3D target detection technology for target detection when performing target detection, because point clouds have the natural advantage of three-dimensional space, and point cloud-based 3D target detection can accurately predict the relative distance of the target from the vehicle. However, the sparsity of point clouds can easily cause the bounding box (bbox) size prediction to be inaccurate during target detection, especially for targets that are far away. SUMMARY

[0003] The present application aims to solve the defects of the prior art, and provides a processing method and device for point cloud target detection based on fused features, an electronic device, and a computer readable storage medium. The method includes the following steps: performing front view feature extraction on the original point cloud, fusing the extracted features with the original point cloud, performing bird's eye view feature extraction on the point cloud fused with the front view features to obtain bird's eye view features fused with the front view features, performing deformable convolution on the bird's eye view features, fusing the convolution results with the bird's eye view features, and performing 3D target detection based on the fused features to obtain corresponding 3D target detection boxes. Through the multi-angle and multi-scale feature extraction and fusion processing of the point cloud, the richness of the point cloud features is enhanced, and the accuracy of the point cloud target detection is improved.

[0004] To achieve the above-mentioned purpose, the first aspect of the present application embodiment provides a processing method for point cloud target detection based on fused features, which comprises the following steps:

[0005] obtaining a first point cloud;

[0006] performing feature extraction processing on the first point cloud based on a front view feature extraction module of a point cloud target detection model to generate a corresponding first front view feature tensor and performing feature fusion processing on the first front view feature tensor and the first point cloud to generate a corresponding first point cloud tensor;

[0007] performing point cloud column feature extraction processing on the first point cloud tensor based on a bird's eye view feature extraction module of the point cloud target detection model to generate a corresponding first point cloud column feature tensor and performing bird's eye view feature extraction processing on the first point cloud column feature tensor to generate a corresponding first bird's eye view feature tensor;

[0008] Based on the feature fusion module of the point cloud target detection model, deformable convolutional feature extraction processing is performed on the first bird’s-eye view feature tensor to generate the corresponding first convolutional feature tensor, and feature fusion processing is performed on the first convolutional feature tensor and the first bird’s-eye view feature tensor to generate the corresponding first bird’s-eye view fused feature tensor.

[0009] The target detection module based on the point cloud target detection model generates a corresponding first detection tensor by performing target detection processing based on the first bird's-eye view fusion feature tensor; the first detection tensor includes multiple first target detection box vectors; the first target detection box vector includes the coordinates of the center point of the first detection box, the 3D size of the first detection box, and the orientation angle of the first detection box.

[0010] Preferably, the first point cloud includes a plurality of first points; each first point corresponds to a set of three-dimensional point cloud coordinates (x, y, z) and a reflection intensity ρ;

[0011] The point cloud target detection model includes a forward-looking feature extraction module, a bird's-eye view feature extraction module, a feature fusion module, and a target detection module; the forward-looking feature extraction module is connected to the bird's-eye view feature extraction module; the bird's-eye view feature extraction module is connected to the feature fusion module; the feature fusion module is connected to the target detection module; the forward-looking feature extraction module includes a forward-looking spatial processing unit, a forward-looking feature extraction unit, and a point cloud feature fusion unit; the input end of the forward-looking spatial processing unit is the model input end, and the output end is connected to the input end of the forward-looking feature extraction unit; the output end of the forward-looking feature extraction unit is connected to the first input end of the point cloud feature fusion unit; the first input end of the point cloud feature fusion unit... The two input terminals are the model input terminals; the bird's-eye view feature extraction module includes a point cloud column feature extraction unit and a bird's-eye view feature extraction unit; the input terminal of the point cloud column feature extraction unit is connected to the output terminal of the point cloud feature fusion unit, and the output terminal is connected to the input terminal of the bird's-eye view feature extraction unit; the feature fusion module includes a deformable convolution unit and a fusion processing unit; the input terminal of the deformable convolution unit is connected to the output terminal of the bird's-eye view feature extraction unit, and the output terminal is connected to the first input terminal of the fusion processing unit; the second input terminal of the fusion processing unit is connected to the output terminal of the bird's-eye view feature extraction unit; the input terminal of the target detection module is connected to the output terminal of the fusion processing unit, and the output terminal is the model output terminal;

[0012] The forward-looking feature extraction unit, the point cloud pillar feature extraction unit, the bird's-eye view feature extraction unit, the deformable convolutional unit, and the target detection module are all implemented based on neural networks. The neural networks corresponding to the forward-looking feature extraction unit and the bird's-eye view feature extraction unit are multi-level feature pyramid networks constructed from specified residual networks, including the ResNet50 network. The neural network corresponding to the point cloud pillar feature extraction unit is a point cloud pillar feature network of the PointPillars model. The neural network corresponding to the deformable convolutional unit is a deformable convolutional network with a kernel size of 3, and the height and width of the input and output tensors of the deformable convolutional network remain unchanged. The neural network corresponding to the target detection module is a 3D target detection bounding box prediction head network composed of a center heatmap head network and a regression head network of the CenterPoint model.

[0013] Preferably, the forward-looking feature extraction module based on the point cloud target detection model performs feature extraction processing on the first point cloud to generate a corresponding first forward-looking feature tensor, and performs feature fusion processing on the first forward-looking feature tensor and the first point cloud to generate a corresponding first point cloud tensor, specifically including:

[0014] The forward-looking spatial processing unit constructs a cylindrical coordinate system with the vehicle as the origin, denoted as the corresponding vehicle cylindrical coordinate system; and sets the height range of the vehicle cylindrical coordinate system as a preset first height range [z]. min ,z max ],z min <0 <z max ; and according to the preset unit angle △θ and unit height △z, the height × width resolution of the front view of the cylindrical coordinate system of the vehicle is set to A×B, where A=(z max -z min B = 360° / Δθ; and according to the coordinate transformation relationship from the point cloud coordinate system to the cylindrical front view pixel coordinate system of the vehicle, the three-dimensional point cloud coordinates (x, y, z) of each first point in the first point cloud are transformed into cylindrical front view pixel coordinates to obtain the corresponding front view pixel coordinates (u, v). On the front view of the cylinder, the pixels corresponding to each front view pixel coordinate (u,v) are recorded as the corresponding first projection points; multiple first projection points with the same front view pixel coordinate (u,v) are grouped into a corresponding set of first projection points on the front view, and in each set of first projection points, the first projection point with the maximum reflection intensity ρ is retained, and all other remaining first projection points are deleted; and the corresponding three-dimensional point cloud coordinates (x,y,z) of each remaining first projection point on the front view of the cylinder are... A first projection point vector of length 4 is formed; and a zero tensor of shape C0×H0×W0 is initialized as the corresponding first front view tensor according to the resolution A×B of the front view of the cylinder; and the first pixel vectors corresponding to each of the remaining first projection points in the first front view tensor are set as the corresponding first projection point vectors; C0, H0, and W0 are the feature dimension, height, and width of the first tensor, respectively, and C0, H0, and W0 are integers greater than zero, C0=4, H0=A, W0=B, and the first tensor includes H0*W0 first pixel vectors;

[0015] The front-view feature extraction unit performs multi-level feature extraction and multi-scale feature fusion processing on the first front-view tensor based on a multi-level feature pyramid network to generate the corresponding first front-view feature tensor. The shape of the first front-view feature tensor is C1×H1×W1; C1, H1, and W1 are the feature dimension, height, and width of the first front-view feature tensor, respectively, and C1, H1, and W1 are integers greater than zero, with C1 defaulting to 16, H1 = H0, and W1 = W0. The first front-view feature tensor includes H1*W1 first front-view feature vectors of length C1. The first front-view feature vectors correspond one-to-one with the first pixel vectors.

[0016] The point cloud feature fusion unit assigns a corresponding all-zero vector of length 4+C1 to each of the first points in the first point cloud as the corresponding first point cloud vector; and sets the first 4 vector data of the corresponding first point cloud vector based on the three-dimensional point cloud coordinates (x, y, z) and the reflection intensity ρ of each of the first points; and iterates through each of the first forward-looking feature vectors of the first forward-looking feature tensor; and during the traversal, the currently traversed first forward-looking feature vector is taken as the corresponding current forward-looking feature vector, and one or more of the first points corresponding to the first projection point of the first pixel vector corresponding to the current forward-looking feature vector and the first projection point corresponding to the first projection point are taken as the corresponding first view pixel coordinates (u, v) as the corresponding first matching points, and the last C1 vector data of the first point cloud vectors corresponding to all the first matching points are set as the C1 vector data of the current forward-looking feature vector; and at the end of the traversal, the first point cloud vectors are combined to form the corresponding first point cloud tensor.

[0017] Preferably, the bird's-eye view feature extraction module based on the point cloud target detection model performs point cloud column feature extraction processing on the first point cloud tensor to generate a corresponding first point cloud column feature tensor, and performs bird's-eye view feature extraction processing on the first point cloud column feature tensor to generate a corresponding first bird's-eye view feature tensor, specifically including:

[0018] The point cloud pillar feature extraction unit, based on the PointPillars model, performs point cloud pillar feature extraction on the first point cloud tensor according to a preset point cloud pillar grid space to generate the corresponding first point cloud pillar feature tensor; the shape of the first point cloud pillar feature tensor is C2×H2×W2, where C2, H2, and W2 are the feature dimension, height, and width of the first point cloud pillar feature tensor, respectively, and C2, H2, and W2 are integers greater than zero;

[0019] The bird's-eye view feature extraction unit performs multi-level feature extraction and multi-scale feature fusion processing on the first point cloud pillar feature tensor based on a multi-level feature pyramid network to generate the corresponding first bird's-eye view feature tensor; the shape of the first bird's-eye view feature tensor is C3×H3×W3, where C3, H3, and W3 are the feature dimension, height, and width of the first bird's-eye view feature tensor, respectively, and C3, H3, and W3 are integers greater than zero, with C3 defaulting to 64.

[0020] Preferably, the feature fusion module based on the point cloud target detection model performs deformable convolutional feature extraction processing on the first bird's-eye view feature tensor to generate a corresponding first convolutional feature tensor, and performs feature fusion processing on the first convolutional feature tensor and the first bird's-eye view feature tensor to generate a corresponding first bird's-eye view fused feature tensor, specifically including:

[0021] The deformable convolutional unit performs feature extraction processing on the first bird's-eye view feature tensor based on the deformable convolutional network to generate the corresponding first convolutional feature tensor; the shape of the first bird's-eye view feature tensor is C3×H3×W3, where C3, H3, and W3 are the feature dimension, height, and width of the first bird's-eye view feature tensor, respectively, and C3, H3, and W3 are integers greater than zero, with C3 defaulting to 64; the shape of the first convolutional feature tensor is C4×H4×W4, where C4, H4, and W4 are the feature dimension, height, and width of the first bird's-eye view feature tensor, respectively, and C4, H4, and W4 are integers greater than zero, where H4 = H3, W4 = W3, and C4 defaults to 64;

[0022] The fusion processing unit performs tensor concatenation on the first convolutional feature tensor and the first bird's-eye view feature tensor according to the feature dimension direction to generate a first concatenated tensor with the shape (C3+C4)×H3×W3; and represents the first concatenated tensor as a second concatenated tensor with the shape H3×W3×(C3+C4); and performs tensor dimensionality reduction processing on the second concatenated tensor based on a preset convolution operator with the shape 1×1×C5 to obtain the corresponding first bird's-eye view fusion feature tensor; the shape of the first bird's-eye view fusion feature tensor is H5×W5×C5, where C5, H5, and W5 are the feature dimension, height, and width of the first bird's-eye view fusion feature tensor, respectively, and C5, H5, and W5 are integers greater than zero, H5 = H3, W5 = W3, and C5 is 64 by default.

[0023] Preferably, the target detection module based on the point cloud target detection model generates a corresponding first detection tensor by performing target detection processing on the first bird's-eye view fusion feature tensor, specifically including:

[0024] The target detection module, based on the center heatmap head network of the 3D target detection box prediction head network, performs target detection box center point coordinate prediction processing based on the first bird's-eye view fusion feature tensor to obtain multiple first detection box center point coordinates; and based on the regression head network of the 3D target detection box prediction head network, performs target detection box height, width, depth, and orientation angle regression prediction processing based on each first detection box center point coordinate and the first bird's-eye view fusion feature tensor to generate corresponding first detection box 3D dimensions and first detection box orientation angles; and each first detection box center point coordinate, corresponding first detection box 3D dimensions, and first detection box orientation angles form a corresponding first target detection box vector; and all obtained first target detection box vectors form a corresponding first detection tensor; the first detection box 3D dimensions include height, width, and depth.

[0025] A second aspect of the present invention provides an apparatus for implementing the point cloud target detection processing method based on fused features as described in the first aspect above. The apparatus includes: an acquisition module, a first feature processing module, a second feature processing module, a third feature processing module, and a 3D detection processing module.

[0026] The acquisition module is used to acquire the first point cloud;

[0027] The first feature processing module is used to perform feature extraction processing on the first point cloud based on the front-look feature extraction module of the point cloud target detection model to generate a corresponding first front-look feature tensor, and to perform feature fusion processing on the first front-look feature tensor and the first point cloud to generate a corresponding first point cloud tensor.

[0028] The second feature processing module is used to perform point cloud column feature extraction processing on the first point cloud tensor based on the bird's-eye view feature extraction module of the point cloud target detection model to generate the corresponding first point cloud column feature tensor, and to perform bird's-eye view feature extraction processing on the first point cloud column feature tensor to generate the corresponding first bird's-eye view feature tensor.

[0029] The third feature processing module is used to perform deformable convolutional feature extraction processing on the first bird's-eye view feature tensor based on the feature fusion module of the point cloud target detection model to generate a corresponding first convolutional feature tensor, and to perform feature fusion processing on the first convolutional feature tensor and the first bird's-eye view feature tensor to generate a corresponding first bird's-eye view fused feature tensor.

[0030] The 3D detection processing module is used to generate a corresponding first detection tensor based on the target detection module of the point cloud target detection model according to the first bird's-eye view fusion feature tensor; the first detection tensor includes multiple first target detection box vectors; the first target detection box vector includes the coordinates of the center point of the first detection box, the 3D size of the first detection box, and the orientation angle of the first detection box.

[0031] A third aspect of the present invention provides an electronic device, including: a memory, a processor, and a transceiver;

[0032] The processor is used to couple with the memory, read and execute instructions in the memory to implement the steps of the method described in the first aspect above;

[0033] The transceiver is coupled to the processor, and the processor controls the transceiver to send and receive messages.

[0034] A fourth aspect of the present invention provides a computer-readable storage medium storing computer instructions that, when executed by a computer, cause the computer to perform the instructions described in the first aspect.

[0035] This invention provides a method, apparatus, electronic device, and computer-readable storage medium for point cloud target detection based on fused features. First, front-view features are extracted from the original point cloud, and the extracted features are fused with the original point cloud. Then, bird's-eye view features are extracted from the point cloud with fused front-view features to obtain bird's-eye view features. Next, deformable convolution is performed on the bird's-eye view features, and the convolution result is fused with the bird's-eye view features. Finally, 3D target detection is performed based on the fused features to obtain multiple corresponding 3D target detection boxes. This invention enhances the feature richness of point clouds and improves the accuracy of point cloud target detection by performing multi-view, multi-scale feature extraction and fusion processing on point clouds. Attached Figure Description

[0036] Figure 1 This is a schematic diagram of a point cloud target detection method based on fused features provided in Embodiment 1 of the present invention;

[0037] Figure 2 This is a block structure diagram of the point cloud target detection model provided in Embodiment 1 of the present invention;

[0038] Figure 3 This is a module structure diagram of a point cloud target detection processing device based on fused features, provided in Embodiment 2 of the present invention.

[0039] Figure 4 This is a schematic diagram of the structure of an electronic device provided in Embodiment 3 of the present invention. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0041] The perception module of the autonomous driving system, based on the point cloud target detection processing method based on fused features provided in Embodiment 1 of the present invention, performs multi-view and multi-scale feature extraction and fusion processing on the point cloud, which can enhance the feature richness of the point cloud and improve the target detection accuracy of the point cloud. Figure 1 This is a schematic diagram of a point cloud target detection processing method based on fused features provided in Embodiment 1 of the present invention, as shown below. Figure 1 As shown, this method mainly includes the following steps:

[0042] Step 1, Obtain the first point cloud;

[0043] The first point cloud includes multiple first points; each first point corresponds to a set of three-dimensional point cloud coordinates (x, y, z) and a reflection intensity ρ.

[0044] Here, the first point cloud is the original point cloud generated by the vehicle-mounted radar scanning the environment around the vehicle. From the publicly available point cloud data structure, we know that the original point cloud, i.e. the first point cloud, is composed of multiple points, i.e., the first point. Each point has a three-dimensional point cloud coordinate (x, y, z) and a reflection intensity ρ.

[0045] After obtaining the first point cloud, the subsequent steps of this invention will perform a series of multi-view, multi-scale feature extraction and fusion processes on the first point cloud based on a point cloud object detection model. Before describing the subsequent steps, the point cloud object detection model of this embodiment of the invention will be described as follows.

[0046] like Figure 2 As shown in the module structure diagram of the point cloud target detection model provided in Embodiment 1 of the present invention, the point cloud target detection model of the present invention includes a forward-looking feature extraction module, a bird's-eye view feature extraction module, a feature fusion module, and a target detection module; the forward-looking feature extraction module is connected to the bird's-eye view feature extraction module; the bird's-eye view feature extraction module is connected to the feature fusion module; the feature fusion module is connected to the target detection module; the forward-looking feature extraction module includes a forward-looking spatial processing unit, a forward-looking feature extraction unit, and a point cloud feature fusion unit; the input end of the forward-looking spatial processing unit is the model input end, and the output end is connected to the input end of the forward-looking feature extraction unit; the output end of the forward-looking feature extraction unit is connected to the first input of the point cloud feature fusion unit. The point cloud feature fusion unit has two input terminals: a point cloud column feature extraction unit and a bird's-eye view feature extraction unit. The input terminal of the point cloud column feature extraction unit is connected to the output terminal of the point cloud feature fusion unit, and the output terminal is connected to the input terminal of the bird's-eye view feature extraction unit. The feature fusion module includes a deformable convolution unit and a fusion processing unit. The input terminal of the deformable convolution unit is connected to the output terminal of the bird's-eye view feature extraction unit, and the output terminal is connected to the first input terminal of the fusion processing unit. The second input terminal of the fusion processing unit is connected to the output terminal of the bird's-eye view feature extraction unit. The input terminal of the target detection module is connected to the output terminal of the fusion processing unit, and the output terminal is the model output terminal.

[0047] It should be noted that the forward-looking feature extraction unit, point cloud pillar feature extraction unit, bird's-eye view feature extraction unit, deformable convolutional unit, and target detection module of the point cloud target detection model implemented in this invention are all based on neural networks. Specifically, the neural networks corresponding to the forward-looking feature extraction unit and the bird's-eye view feature extraction unit are multi-level feature pyramid networks (FPNs) constructed from specified residual networks (ResNet), including the ResNet50 network. The neural network corresponding to the point cloud pillar feature extraction unit is the Pillar Feature Net of the PointPillars model. The neural network corresponding to the deformable convolutional unit is a Deformable Convolutional Network (DCN), with a kernel size of 3, and the height and width of the input and output tensors remain unchanged. The neural network corresponding to the target detection module is a 3D target detection bounding box prediction head network composed of a Center heatmap head network and a Regression heads network from the CenterPoint model.The detailed network structures of ResNet and ResNet50 in the above schemes can be found in the paper "Deep Residual Learning for Image Recognition," and the detailed network structure of FPN can be found in the paper "Feature Pyramid Networks for Object Detection." Feature extraction based on the ResNet (ResNet50) + FPN neural network structure can not only provide multi-scale feature information but also fully fuse features at each scale through multi-level upsampling, thereby increasing feature richness. The detailed network structure of the PointPillars model can be found in the paper "PointPillars: Fast Encoders for Object Detection from Point Clouds," and the Pillar Feature based on the PointPillars model... Net can perform Pillar clustering on the input point cloud based on a defined Pillar grid network (e.g., Pillar grid height of 10 meters, Pillar grid plane of 0.1 meters * 0.1 meters) and extract the point cloud features in each Pillar grid to obtain the corresponding point cloud pillar features, thus obtaining the corresponding point cloud bird's eye view (BEV) features. Detailed network structure of DCN can be found in the papers "Deformable Convolutional Networks" and "Deformableconvnets v2: More deformable, better results". Feature extraction based on DCN can obtain more feature information than conventional convolutional networks. Detailed network structure of the CenterPoint model can be found in the paper "Center-based 3D Object Detection and Tracking". The Center heatmap head based on the CenterPoint model can predict the center point coordinates of each object, and the Regression heads based on the CenterPoint model can regress and predict the height, width, depth, and orientation angle of the object detection box based on the center point coordinates of each object.

[0048] Step 2: Based on the point cloud target detection model, the front-looking feature extraction module performs feature extraction processing on the first point cloud to generate the corresponding first front-looking feature tensor, and performs feature fusion processing on the first front-looking feature tensor and the first point cloud to generate the corresponding first point cloud tensor.

[0049] Here, in the current step of this embodiment of the invention, the point cloud target detection model first extracts front view features from the original point cloud, i.e., the first point cloud, and then fuses the extracted features with the first point cloud to obtain a point cloud tensor that has been fused with front view features, i.e., the first point cloud tensor.

[0050] Specifically, this includes: Step 21, where the forward-looking spatial processing unit constructs a cylindrical coordinate system with the vehicle as the origin, denoted as the corresponding vehicle cylindrical coordinate system; and sets the height range of the vehicle cylindrical coordinate system to a preset first height range [z]. min ,z max ],z min <0 <z max ; and according to the preset unit angle △θ and unit height △z, set the height × width resolution of the front view (FV) of the vehicle cylindrical coordinate system to A×B, where A=(z max -Z min B = 360° / Δθ; and based on the coordinate transformation relationship from the point cloud coordinate system to the cylindrical front view pixel coordinate system of the vehicle, the three-dimensional point cloud coordinates (x, y, z) of each first point in the first point cloud are transformed into the cylindrical front view pixel coordinates to obtain the corresponding front view pixel coordinates (u, v). On the front view of the cylinder, the pixels corresponding to each front view pixel coordinate (u,v) are recorded as the corresponding first projection points; multiple first projection points with the same front view pixel coordinate (u,v) are grouped into a set of corresponding first projection points, and the first projection point with the maximum reflection intensity ρ is retained in each set, while all other remaining first projection points are deleted; and the corresponding 3D point cloud coordinates (x,y,z) of each remaining first projection point on the front view of the cylinder are used to determine the first projection point. Form a first projection point vector with a corresponding length of 4; and initialize a zero tensor with a shape of C0×H0×W0 as the corresponding first front view tensor according to the resolution A×B of the front view of the cylinder; and set the first pixel vector in the first front view tensor corresponding to each of the remaining first projection points as the corresponding first projection point vector.

[0051] Where C0, H0, and W0 are the feature dimension, height, and width of the first tensor, respectively, and C0, H0, and W0 are integers greater than zero, C0 = 4, H0 = A, and W0 = B. The first tensor includes H0 * W0 first pixel vectors.

[0052] Step 22: The front view feature extraction unit performs multi-level feature extraction and multi-scale feature fusion processing on the first front view tensor based on a multi-level feature pyramid network to generate the corresponding first front view feature tensor.

[0053] The first forward-looking feature tensor has a shape of C1×H1×W1; C1, H1, and W1 are the feature dimension, height, and width of the first forward-looking feature tensor, respectively. C1, H1, and W1 are integers greater than zero, with C1 defaulting to 16, H1 = H0, and W1 = W0. The first forward-looking feature tensor includes H1*W1 first forward-looking feature vectors of length C1. Each first forward-looking feature vector corresponds one-to-one with a first pixel vector.

[0054] Step 23: The point cloud feature fusion unit assigns a corresponding zero vector of length 4+C1 to each first point of the first point cloud as the corresponding first point cloud vector; and sets the first 4 vector data of the corresponding first point cloud vector based on the three-dimensional point cloud coordinates (x,y,z) and reflection intensity ρ of each first point; and iterates through each first forward-looking feature vector of the first forward-looking feature tensor; during the traversal, the currently traversed first forward-looking feature vector is taken as the corresponding current forward-looking feature vector, and one or more first points corresponding to the front view pixel coordinates (u,v) of the first projection point corresponding to the first pixel vector of the current forward-looking feature vector are taken as the corresponding first matching points, and the last C1 vector data of the first point cloud vectors corresponding to all the first matching points are set as the C1 vector data of the current forward-looking feature vector; and at the end of the traversal, the first point cloud vectors are composed of all the first point cloud vectors to form the corresponding first point cloud tensor.

[0055] Step 3: The bird's-eye view feature extraction module based on the point cloud target detection model performs point cloud column feature extraction processing on the first point cloud tensor to generate the corresponding first point cloud column feature tensor, and performs bird's-eye view feature extraction processing on the first point cloud column feature tensor to generate the corresponding first bird's-eye view feature tensor.

[0056] Here, in the current step, the point cloud target detection model of this embodiment of the invention extracts bird's-eye view features from the first point cloud tensor that incorporates front view features, thereby obtaining the bird's-eye view features that incorporate the front view features, namely the first bird's-eye view feature tensor.

[0057] Specifically, it includes: Step 31, where the point cloud pillar feature extraction unit performs point cloud pillar feature extraction processing on the first point cloud tensor according to the preset point cloud pillar grid space based on the point cloud pillar feature network of the PointPillars model to generate the corresponding first point cloud pillar feature tensor.

[0058] The shape of the first point cloud pillar feature tensor is C2×H2×W2, where C2, H2, and W2 are the feature dimension, height, and width of the first point cloud pillar feature tensor, respectively, and C2, H2, and W2 are integers greater than zero.

[0059] Here, the preset point cloud column grid space can be set according to real-time needs, such as setting the point cloud column grid space to consist of multiple point cloud column grids, with each point cloud column grid having a height of 10 meters and a grid plane of 0.1 meters * 0.1 meters.

[0060] Step 32: The bird's-eye view feature extraction unit performs multi-level feature extraction and multi-scale feature fusion processing on the first point cloud pillar feature tensor based on a multi-level feature pyramid network to generate the corresponding first bird's-eye view feature tensor.

[0061] The first bird's-eye view feature tensor has a shape of C3×H3×W3, where C3, H3, and W3 are the feature dimension, height, and width of the first bird's-eye view feature tensor, respectively. C3, H3, and W3 are integers greater than zero, and C3 is 64 by default.

[0062] Step 4: The feature fusion module based on the point cloud target detection model performs deformable convolution feature extraction processing on the first bird's-eye view feature tensor to generate the corresponding first convolution feature tensor, and performs feature fusion processing on the first convolution feature tensor and the first bird's-eye view feature tensor to generate the corresponding first bird's-eye view fused feature tensor.

[0063] Here, in the current step, the point cloud target detection model of this embodiment of the invention performs deformable convolution on the bird's-eye view feature, i.e., the first bird's-eye view feature tensor, which is fused with the forward-looking features, and fuses the convolution result with the bird's-eye view feature. The feature richness can be further enhanced through the current step.

[0064] Specifically, it includes: Step 41, where the deformable convolutional unit performs feature extraction processing on the first bird's-eye view feature tensor based on the deformable convolutional network to generate the corresponding first convolutional feature tensor;

[0065] The first bird's-eye view feature tensor has a shape of C3×H3×W3, where C3, H3, and W3 are the feature dimensions, height, and width of the first bird's-eye view feature tensor, respectively. C3, H3, and W3 are integers greater than zero, and C3 defaults to 64. The first convolutional feature tensor has a shape of C4×H4×W4, where C4, H4, and W4 are the feature dimensions, height, and width of the first bird's-eye view feature tensor, respectively. C4, H4, and W4 are integers greater than zero, where H4 = H3, W4 = W3, and C4 defaults to 64.

[0066] Step 42: The fusion processing unit performs tensor concatenation on the first convolutional feature tensor and the first bird's-eye view feature tensor according to the feature dimension direction to generate a first concatenated tensor with the shape (C3+C4)×H3×W3; and represents the first concatenated tensor as a second concatenated tensor with the shape H3×W3×(C3+C4); and performs tensor dimensionality reduction on the second concatenated tensor based on a preset convolution operator with the shape 1×1×C5 to obtain the corresponding first bird's-eye view fusion feature tensor.

[0067] The first bird's-eye view fusion feature tensor has a shape of H5×W5×C5, where C5, H5, and W5 are the feature dimension, height, and width of the first bird's-eye view fusion feature tensor, respectively. C5, H5, and W5 are integers greater than zero, H5 = H3, W5 = W3, and C5 is 64 by default.

[0068] Step 5: The target detection module based on the point cloud target detection model performs target detection processing based on the first bird's-eye view fusion feature tensor to generate the corresponding first detection tensor;

[0069] The first detection tensor includes multiple first target detection box vectors; each first target detection box vector includes the coordinates of the center point of the first detection box, the 3D size of the first detection box, and the orientation angle of the first detection box; the 3D size of the first detection box includes height, width, and depth.

[0070] Specifically, this includes: the target detection module, based on the center heatmap head network of the 3D target detection box prediction head network, performs target detection box center point coordinate prediction processing based on the first bird's-eye view fusion feature tensor to obtain multiple first detection box center point coordinates; and based on the regression head network of the 3D target detection box prediction head network, performs target detection box height, width, depth, and orientation angle regression prediction processing based on the center point coordinates of each first detection box and the first bird's-eye view fusion feature tensor to generate corresponding first detection box 3D dimensions and first detection box orientation angles; and the center point coordinates of each first detection box and the corresponding first detection box 3D dimensions and first detection box orientation angles form a corresponding first target detection box vector; and all the obtained first target detection box vectors form a corresponding first detection tensor.

[0071] Figure 3 This is a module structure diagram of a point cloud target detection processing device based on fused features provided in Embodiment 2 of the present invention. This device can be a terminal device or server implementing the aforementioned method embodiments, or it can be a device that enables the aforementioned terminal device or server to implement the aforementioned method embodiments. For example, the device can be a device or chip system of the aforementioned terminal device or server. Figure 3 As shown, the device includes: an acquisition module 201, a first feature processing module 202, a second feature processing module 203, a third feature processing module 204, and a 3D detection processing module 205.

[0072] The acquisition module 201 is used to acquire the first point cloud.

[0073] The first feature processing module 202 is used to perform feature extraction processing on the first point cloud based on the point cloud target detection model's front-look feature extraction module to generate a corresponding first front-look feature tensor, and to perform feature fusion processing on the first front-look feature tensor and the first point cloud to generate a corresponding first point cloud tensor.

[0074] The second feature processing module 203 is used by the bird's-eye view feature extraction module based on the point cloud target detection model to perform point cloud column feature extraction processing on the first point cloud tensor to generate the corresponding first point cloud column feature tensor, and to perform bird's-eye view feature extraction processing on the first point cloud column feature tensor to generate the corresponding first bird's-eye view feature tensor.

[0075] The third feature processing module 204 is used by the feature fusion module of the point cloud target detection model to perform deformable convolutional feature extraction processing on the first bird's-eye view feature tensor to generate the corresponding first convolutional feature tensor, and to perform feature fusion processing on the first convolutional feature tensor and the first bird's-eye view feature tensor to generate the corresponding first bird's-eye view fused feature tensor.

[0076] The 3D detection processing module 205 is used by the target detection module based on the point cloud target detection model to generate a corresponding first detection tensor by performing target detection processing based on the first bird's-eye view fusion feature tensor; the first detection tensor includes multiple first target detection box vectors; the first target detection box vectors include the coordinates of the center point of the first detection box, the 3D size of the first detection box, and the orientation angle of the first detection box.

[0077] The processing device for point cloud target detection based on fusion features provided in this embodiment of the invention can execute the method steps in the above method embodiment. Its implementation principle and technical effect are similar, and will not be repeated here.

[0078] It should be noted that the division of the various modules in the above device is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, these modules can be implemented entirely in software via processing element calls; they can be fully implemented in hardware; or some modules can be implemented by processing element calls to software, while others are implemented in hardware. For example, the acquisition module can be a separate processing element, or it can be integrated into a chip in the above device. Alternatively, it can be stored as program code in the memory of the above device, and called and executed by a processing element of the device. The implementation of other modules is similar. Moreover, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each of the above modules can be completed through integrated logic circuits in the hardware of the processor element or through software instructions.

[0079] For example, these modules can be one or more integrated circuits configured to implement the above methods, such as one or more Application Specific Integrated Circuits (ASICs), one or more Digital Signal Processors (DSPs), or one or more Field Programmable Gate Arrays (FPGAs). As another example, when a module is implemented using processing element scheduler code, the processing element can be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. Furthermore, these modules can be integrated together as a System-on-a-Chip (SOC).

[0080] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. This computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the foregoing method embodiments are generated. The computer described above can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The aforementioned computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the aforementioned computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, Bluetooth, microwave, etc.) means. The aforementioned computer-readable storage medium can be any available medium that a computer can access, or a data storage device such as a server or data center that integrates one or more available media. The aforementioned available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state disks (SSDs)).

[0081] Figure 4 This is a schematic diagram of an electronic device provided in Embodiment 3 of the present invention. This electronic device can be the aforementioned terminal device or server, or it can be a terminal device or server connected to the aforementioned terminal device or server that implements the method of the embodiments of the present invention. Figure 4As shown, the electronic device may include: a processor 301 (e.g., CPU), a memory 302, and a transceiver 303; the transceiver 303 is coupled to the processor 301, and the processor 301 controls the transmission and reception operations of the transceiver 303. The memory 302 may store various instructions for performing various processing functions and implementing the processing steps described in the foregoing method embodiments. Preferably, the electronic device involved in the embodiments of the present invention further includes: a power supply 304, a system bus 305, and a communication port 306. The system bus 305 is used to realize communication connections between components. The communication port 306 is used for communication between the electronic device and other peripherals.

[0082] exist Figure 4 The system bus 305 mentioned can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This system bus can be divided into address bus, data bus, control bus, etc. For ease of representation, it is represented by only one thick line in the figure, but this does not indicate that there is only one bus or one type of bus. The communication interface is used to enable communication between the database access device and other devices (e.g., clients, read-write libraries, and read-only libraries). Memory may include Random Access Memory (RAM) and may also include non-volatile memory, such as at least one disk storage device.

[0083] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), graphics processing units (GPUs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0084] It should be noted that the embodiments of the present invention also provide a computer-readable storage medium storing instructions that, when run on a computer, cause the computer to perform the methods and processes provided in the above embodiments.

[0085] This invention also provides a chip for executing instructions, which is used to perform the processing steps described in the foregoing method embodiments.

[0086] This invention provides a method, apparatus, electronic device, and computer-readable storage medium for point cloud target detection based on fused features. First, front-view features are extracted from the original point cloud, and the extracted features are fused with the original point cloud. Then, bird's-eye view features are extracted from the point cloud with fused front-view features to obtain bird's-eye view features. Next, deformable convolution is performed on the bird's-eye view features, and the convolution result is fused with the bird's-eye view features. Finally, 3D target detection is performed based on the fused features to obtain multiple corresponding 3D target detection boxes. This invention enhances the feature richness of point clouds and improves the accuracy of point cloud target detection by performing multi-view, multi-scale feature extraction and fusion processing on point clouds.

[0087] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0088] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented in hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0089] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A processing method for point cloud target detection based on fusion features, characterized in that, The method comprises: acquiring a first point cloud; performing feature extraction processing on the first point cloud based on a front-view feature extraction module of a point cloud target detection model to generate a corresponding first front-view feature tensor and performing feature fusion processing on the first front-view feature tensor and the first point cloud to generate a corresponding first point cloud tensor; performing point cloud column feature extraction processing on the first point cloud tensor based on an aerial view feature extraction module of the point cloud target detection model to generate a corresponding first point cloud column feature tensor and performing aerial view feature extraction processing on the first point cloud column feature tensor to generate a corresponding first aerial view feature tensor; performing deformable convolution feature extraction processing on the first aerial view feature tensor based on a feature fusion module of the point cloud target detection model to generate a corresponding first convolution feature tensor and performing feature fusion processing on the first convolution feature tensor and the first aerial view feature tensor to generate a corresponding first aerial view fusion feature tensor; performing target detection processing on the first aerial view fusion feature tensor based on a target detection module of the point cloud target detection model to generate a corresponding first detection tensor; the first detection tensor comprises a plurality of first target detection box vectors; the first target detection box vector comprises a first detection box center point coordinate, a first detection box 3D size and a first detection box orientation angle.

2. The processing method for point cloud target detection based on fusion features according to claim 1, characterized in that the first point cloud comprises a plurality of first points; each first point corresponds to a set of three-dimensional point cloud coordinates (x, y, z) and a reflection intensity p; the point cloud target detection model comprises the front-view feature extraction module, the aerial view feature extraction module, the feature fusion module and the target detection module; the front-view feature extraction module is connected with the aerial view feature extraction module; the aerial view feature extraction module is connected with the feature fusion module; the feature fusion module is connected with the target detection module; the front-view feature extraction module comprises a front-view space processing unit, a front-view feature extraction unit and a point cloud feature fusion unit; the input end of the front-view space processing unit is a model input end, and the output end is connected with the input end of the front-view feature extraction unit; the output end of the front-view feature extraction unit is connected with the first input end of the point cloud feature fusion unit; the second input end of the point cloud feature fusion unit is the model input end; the aerial view feature extraction module comprises a point cloud column feature extraction unit and an aerial view feature extraction unit; the input end of the point cloud column feature extraction unit is connected with the output end of the point cloud feature fusion unit, and the output end is connected with the input end of the aerial view feature extraction unit; the feature fusion module comprises a deformable convolution unit and a fusion processing unit; the input end of the deformable convolution unit is connected with the output end of the aerial view feature extraction unit, and the output end is connected with the first input end of the fusion processing unit; the second input end of the fusion processing unit is connected with the output end of the aerial view feature extraction unit; the input end of the target detection module is connected with the output end of the fusion processing unit, and the output end is a model output end; The front-view feature extraction unit, the point cloud column feature extraction unit, the bird's eye view feature extraction unit, the deformable convolution unit and the target detection module are all realized based on neural networks; the neural networks corresponding to the front-view feature extraction unit and the bird's eye view feature extraction unit are both multi-level feature pyramid networks constructed by a specified residual network, and the specified residual network includes a resnet50 network; the neural network corresponding to the point cloud column feature extraction unit is a point cloud column feature network of a PointPillars model; the neural network corresponding to the deformable convolution unit is a deformable convolution network, the convolution kernel size of the deformable convolution network is 3, and the height and width of the input and output tensors of the deformable convolution network are invariant; the neural network corresponding to the target detection module is a 3D target detection box prediction head network composed of a center heat map head network and a regression head network of a CenterPoint model.

3. The processing method for point cloud target detection based on fusion features according to claim 2, characterized in that, The front-view feature extraction module based on the point cloud target detection model performs feature extraction processing on the first point cloud to generate a corresponding first front-view feature tensor and performs feature fusion processing on the first front-view feature tensor and the first point cloud to generate a corresponding first point cloud tensor, specifically including: A cylindrical coordinate system with the ego vehicle as the coordinate origin is constructed by the front-view spatial processing unit, denoted as a corresponding ego vehicle cylindrical coordinate system; and the height range of the ego vehicle cylindrical coordinate system is set as a preset first height range [z min ,z max ],z min <0<z max ; and the high*wide resolution of the cylindrical front view of the ego vehicle cylindrical coordinate system is set as A*B according to a preset unit angle △θ and unit height △z, A=(z max -z min ) / △z, B=360° / △θ; and a cylindrical front view pixel coordinate conversion process is performed on the three-dimensional point cloud coordinates (x, y, z) of each first point in the first point cloud according to the coordinate conversion relationship from the point cloud coordinate system to the pixel coordinate system of the cylindrical front view of the ego vehicle cylindrical coordinate system, to obtain a corresponding front view pixel coordinate (u, v), and each first projection point corresponding to the front view pixel coordinate (u, v) is recorded as a corresponding first projection point on the cylindrical front view; a corresponding first projection point set is formed by a plurality of first projection points with the same front view pixel coordinate (u, v) on the cylindrical front view, and the first projection point with the maximum corresponding reflection intensity ρ is retained and all other remaining first projection points are deleted in each first projection point set; and a corresponding first projection point vector with a length of 4 is formed by the three-dimensional point cloud coordinates (x, y, z) corresponding to each remaining first projection point on the cylindrical front view and ; a full-zero tensor with a shape of C0*H0*W0 is initialized as a corresponding first front view tensor according to the resolution A*B of the cylindrical front view; and the first pixel vector corresponding to each remaining first projection point in the first front view tensor is set as the corresponding first projection point vector; C0, H0, and W0 are the feature dimension, height, and width of the first front view tensor respectively, C0, H0, and W0 are integers greater than zero, C0=4, H0=A, and W0=B, and the first front view tensor includes H0*W0 first pixel vectors; The front-view feature extraction unit performs multi-level feature extraction and multi-scale feature fusion processing on the first front-view image tensor based on a multi-level feature pyramid network to generate a corresponding first front-view feature tensor; the shape of the first front-view feature tensor is C1×H1×W1; C1, H1 and W1 are respectively the feature dimension, height and width of the first front-view feature tensor, and C1, H1 and W1 are integers greater than zero, C1 is 16 by default, H1=H0 and W1=W0; the first front-view feature tensor includes H1*W1 first front-view feature vectors with a length of C1; the first front-view feature vector corresponds to the first pixel vector one by one; The point cloud feature fusion unit assigns each first point of the first point cloud a corresponding all-zero vector with a length of 4+C1 as a corresponding first point cloud vector; and sets the first 4 vector data of the corresponding first point cloud vector based on the three-dimensional point cloud coordinates (x, y, z) and the reflection intensity p of each first point; and traverses each first front-view feature vector of the first front-view feature tensor; and during traversal, the first front-view feature vector currently traversed is taken as a corresponding current front-view feature vector, one or more first points corresponding to the front-view image pixel coordinates (u, v) of the first projection point corresponding to the first pixel vector corresponding to the current front-view feature vector are taken as corresponding first matching points, and the last C1 vector data of the first point cloud vector corresponding to all the first matching points are set as the C1 vector data of the current front-view feature vector; and at the end of traversal, the first point cloud tensor corresponding to all the first point cloud vectors is generated. The front-view feature extraction module based on the point cloud target detection model performs feature extraction processing on the first point cloud to generate a corresponding first front-view feature tensor and performs feature fusion processing on the first front-view feature tensor and the first point cloud to generate a corresponding first point cloud tensor, specifically including: The front-view feature extraction unit performs multi-level feature extraction and multi-scale feature fusion processing on the first front-view image tensor based on a multi-level feature pyramid network to generate a corresponding first front-view feature tensor; the shape of the first front-view feature tensor is C1×H1×W1; C1, H1 and W1 are respectively the feature dimension, height and width of the first front-view feature tensor, and C1, H1 and W1 are integers greater than zero, C1 is 16 by default, H1=H0 and W1=W0; the first front-view feature tensor includes H1*W1 first front-view feature vectors with a length of C1; the first front-view feature vector corresponds to the first pixel vector one by one; The point cloud feature fusion unit assigns each first point of the first point cloud a corresponding all-zero vector with a length of 4+C1 as a corresponding first point cloud vector; and sets the first 4 vector data of the corresponding first point cloud vector based on the three-dimensional point cloud coordinates (x, y, z) and the reflection intensity p of each first point; and traverses each first front-view feature vector of the first front-view feature tensor; and during traversal, the first front-view feature vector currently traversed is taken as a corresponding current front-view feature vector, one or more first points corresponding to the front-view image pixel coordinates (u, v) of the first projection point corresponding to the first pixel vector corresponding to the current front-view feature vector are taken as corresponding first matching points, and the last C1 vector data of the first point cloud vector corresponding to all the first matching points are set as the C1 vector data of the current front-view feature vector; and at the end of traversal, the first point cloud tensor corresponding to all the first point cloud vectors is generated.

4. The processing method for point cloud target detection based on fusion features according to claim 2, characterized in that, The bird's eye view feature extraction module based on the point cloud target detection model performs point cloud column feature extraction processing on the first point cloud tensor to generate a corresponding first point cloud column feature tensor and performs bird's eye view feature extraction processing on the first point cloud column feature tensor to generate a corresponding first bird's eye view feature tensor, specifically including: The point cloud column feature extraction unit performs point cloud column feature extraction processing on the first point cloud tensor based on a point cloud column feature network of a PointPillars model according to a preset point cloud column grid space to generate the corresponding first point cloud column feature tensor; the shape of the first point cloud column feature tensor is C2×H2×W2, C2, H2 and W2 are respectively a feature dimension, a height and a width of the first point cloud column feature tensor, and C2, H2 and W2 are integers greater than zero; The bird's eye view feature extraction unit performs multi-level feature extraction and multi-scale feature fusion processing on the first point cloud column feature tensor based on a multi-level feature pyramid network to generate the corresponding first bird's eye view feature tensor; the shape of the first bird's eye view feature tensor is C3×H3×W3, C3, H3 and W3 are respectively a feature dimension, a height and a width of the first bird's eye view feature tensor, and C3, H3 and W3 are integers greater than zero, and C3 is 64 by default.

5. The processing method for point cloud target detection based on fusion features according to claim 2, characterized in that, The feature fusion module based on the point cloud target detection model performs deformable convolution feature extraction processing on the first bird's eye view feature tensor to generate a corresponding first convolution feature tensor and performs feature fusion processing on the first convolution feature tensor and the first bird's eye view feature tensor to generate a corresponding first bird's eye view fusion feature tensor, specifically including: The deformable convolution unit performs feature extraction processing on the first bird's eye view feature tensor based on a deformable convolution network to generate the corresponding first convolution feature tensor; the shape of the first bird's eye view feature tensor is C3×H3×W3, C3, H3 and W3 are respectively a feature dimension, a height and a width of the first bird's eye view feature tensor, and C3, H3 and W3 are integers greater than zero, and C3 is 64 by default; the shape of the first convolution feature tensor is C4×H4×W4, C4, H4 and W4 are respectively a feature dimension, a height and a width of the first bird's eye view feature tensor, and C4, H4 and W4 are integers greater than zero, H4=H3, W4=W3, and C4 is 64 by default; The fusion processing unit is configured to perform tensor splicing on the first convolution feature tensor and the first bird's eye view feature tensor in a feature dimension direction to generate a first spliced tensor with a shape of (C3+C4)×H3×W3, represent the first spliced tensor as a second spliced tensor with a shape of H3×W3×(C3+C4), and perform tensor dimension reduction processing on the second spliced tensor based on a preset convolution operator with a shape of 1×1×C5 to obtain the first bird's eye view fusion feature tensor.

6. The processing method for point cloud target detection based on fusion features according to claim 2, characterized in that, The target detection module is configured to perform target detection processing on the first bird's eye view fusion feature tensor to generate a first detection tensor. The target detection module is configured to perform target detection frame center point coordinate prediction processing on the first bird's eye view fusion feature tensor based on a center heat map head network of a 3D target detection frame prediction head network to obtain a plurality of first detection frame center point coordinates, perform height, width, depth, and orientation angle regression prediction processing on each first detection frame center point coordinate and the first bird's eye view fusion feature tensor based on a regression head network of the 3D target detection frame prediction head network to generate a first detection frame 3D size and a first detection frame orientation angle, and compose a first target detection frame vector from each first detection frame center point coordinate, the first detection frame 3D size, and the first detection frame orientation angle.

7. An apparatus for performing the processing method of point cloud object detection based on fusion features according to any one of claims 1-6, characterized in that, The device comprises an acquisition module, a first feature processing module, a second feature processing module, a third feature processing module, and a 3D detection processing module. The acquisition module is configured to acquire a first point cloud. The first feature processing module is configured to perform feature extraction processing on the first point cloud based on a front view feature extraction module of a point cloud target detection model to generate a first front view feature tensor, and perform feature fusion processing on the first front view feature tensor and the first point cloud to generate a first point cloud tensor. The second feature processing module is configured to perform point cloud column feature extraction processing on the first point cloud tensor based on an aerial view feature extraction module of the point cloud target detection model to generate a first point cloud column feature tensor, and perform aerial view feature extraction processing on the first point cloud column feature tensor to generate a first bird's eye view feature tensor. The third feature processing module is configured to perform deformable convolution feature extraction processing on the first bird's eye view feature tensor based on a feature fusion module of the point cloud target detection model to generate a first convolution feature tensor, and perform feature fusion processing on the first convolution feature tensor and the first bird's eye view feature tensor to generate a first bird's eye view fusion feature tensor. The third feature processing module is configured to perform deformable convolution feature extraction processing on the first bird's eye view feature tensor based on a feature fusion module of the point cloud target detection model to generate a first convolution feature tensor, and perform feature fusion processing on the first convolution feature tensor and the first bird's eye view feature tensor to generate a first bird's eye view fusion feature tensor. The 3D detection processing module is configured to perform target detection processing based on a target detection module of the point cloud target detection model to generate a corresponding first detection tensor according to the first bird's eye fusion feature tensor; the first detection tensor includes a plurality of first target detection box vectors; and each first target detection box vector includes a first detection box center point coordinate, a first detection box 3D size, and a first detection box orientation angle.

8. An electronic device, comprising: Comprise: a memory, a processor and a transceiver; the processor is used for coupling with the memory, reading and executing instructions in the memory to realize the method steps of any one of claims 1-6; the transceiver is coupled with the processor, and the transceiver is controlled by the processor to perform message transceiving.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions, when the computer instructions are executed by a computer, the computer instructions make the computer execute the instructions of the method of any one of claims 1-6.

Images