Point cloud target detection methods, devices, electronic equipment and storage media

Abstract

Embodiments of the present application provide a point cloud target detection method and device, electronic equipment and storage medium, wherein the method comprises: collecting three-dimensional point cloud data and two-dimensional image data through a radar and a camera; extracting 3D point-level features and 3D voxel features from the three-dimensional point cloud data through a 3D sparse point feature extraction module and a 3D sparse voxel feature extraction module; extracting 2D perspective features of an initial perspective view through a 2D feature extraction module; extracting 2D pixel features from normalized image data through the 2D feature extraction module; fusing the 3D point-level features, the 3D voxel features, the 2D perspective features and the 2D pixel features through a multi-modal feature fusion module to obtain final fusion features; and calculating the final fusion features through a prediction module to obtain three-dimensional information of an obstacle target. In this way, multiple features are deeply fused, and the final fusion features are used to predict the 3D bounding box of the obstacle target in the scene, thereby achieving a more robust detection effect.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a point cloud target detection method, apparatus, electronic device, and storage medium. Background Technology

[0002] Existing point cloud object detection schemes include those based on raw point processing, those based on mesh voxelization, and those based on 2D views. These common schemes process point cloud data according to a specific form, which can easily lead to insufficient representation of point cloud features. In addition, there are point cloud and 2D image fusion schemes based on single-modal processing. These schemes have relatively poor feature fusion effects during the fusion of images and point clouds, failing to fundamentally increase the richness of features to uncover the true data features of point clouds and images. Summary of the Invention

[0003] To address the aforementioned technical problems, embodiments of the present invention provide a point cloud target detection method, apparatus, electronic device, and storage medium.

[0004] In a first aspect, embodiments of the present invention provide a point cloud target detection method, comprising:

[0005] Three-dimensional point cloud data is acquired by radar, and two-dimensional images are acquired by camera to obtain the mapping relationship between the three-dimensional point cloud data and the two-dimensional images.

[0006] 3D point-level features are extracted from the 3D point cloud data using a 3D sparse point feature extraction module.

[0007] 3D voxel features are extracted from the 3D point cloud data using a 3D sparse voxel feature extraction module.

[0008] The three-dimensional point cloud data is projected from a radar perspective to obtain an initial perspective view. The 2D perspective features of the initial perspective view are then extracted using a 2D feature extraction module.

[0009] The two-dimensional image is normalized to obtain normalized image data, and 2D pixel features are extracted from the normalized image data by the 2D feature extraction module.

[0010] The multimodal feature fusion module fuses the 3D point-level features, the 3D voxel features, the 2D perspective features, and the 2D pixel features according to the mapping relationship to obtain the final fused features;

[0011] The prediction module calculates the final fused features to obtain the three-dimensional information of the obstacle target.

[0012] According to a specific embodiment of this disclosure, the step of obtaining the mapping relationship between the three-dimensional point cloud data and the two-dimensional image includes:

[0013] The mapping relationship between the pixels of the two-dimensional image and the three-dimensional points of the three-dimensional point cloud data is determined based on the calibration parameters between the radar and the camera.

[0014] According to a specific embodiment of this disclosure, the step of extracting 3D point-level features from the 3D point cloud data using a 3D sparse point feature extraction module includes:

[0015] The 3D sparse point feature extraction module performs fixed-point downsampling on the 3D point cloud data to obtain a fixed number of sparse point clouds, which serve as effective 3D point cloud data to characterize the current scene.

[0016] The features of sparse point clouds obtained through point cloud downsampling, point cloud grouping, feature extraction, and feature aggregation are used as 3D point-level features of the effective 3D point cloud data of the scene.

[0017] According to a specific embodiment of this disclosure, the step of extracting 3D voxel features from the 3D point cloud data using a 3D sparse voxel feature extraction module includes:

[0018] Extract the region of interest (ROI) point cloud from the 3D point cloud data;

[0019] The point cloud of the region of interest is divided into voxels to obtain an effective three-dimensional sparse voxel point cloud. The attributes of the effective three-dimensional sparse voxel point cloud include initial features and three-dimensional index coordinates in space. The initial features are the average of three coordinate values ​​and reflectance values ​​of all points in each three-dimensional voxel.

[0020] The 3D sparse voxel feature extraction module extracts 3D voxel features from the effective 3D sparse voxel point cloud.

[0021] According to a specific embodiment of this disclosure, the step of extracting 3D voxel features from the effective 3D sparse voxel point cloud using the 3D sparse voxel feature extraction module includes:

[0022] The mean value of the three-dimensional coordinate information of each three-dimensional point of the effective three-dimensional sparse voxel point cloud is determined, and the mean value of the reflectance of each three-dimensional point of the effective three-dimensional sparse voxel point cloud is determined. The mean value of the three-dimensional coordinate information and the mean value of the reflectance are used as the initial 3D voxel features of the effective three-dimensional sparse voxel point cloud, and the initial 3D voxel features are input into the 3D sparse voxel feature extraction module.

[0023] The 3D sparse voxel feature extraction module determines the feature output coordinate information based on the 3D coordinate information of the effective 3D sparse voxel point cloud, and performs convolution operation on the initialized voxel features based on the 3D coordinate information of the effective 3D sparse voxel point cloud and the feature output coordinate information to obtain the 3D voxel features.

[0024] According to a specific embodiment of this disclosure, the step of performing perspective projection on the three-dimensional point cloud data according to a radar perspective to obtain an initial perspective view includes:

[0025] Calculate the Euclidean distance and elevation angle between each three-dimensional point in the three-dimensional point cloud data and the radar;

[0026] Based on the Euclidean distance, pitch angle, wiring harness, and 3D coordinate information of each 3D point, map each 3D point onto a preset 2D perspective view to obtain the initial perspective view.

[0027] According to a specific embodiment of this disclosure, the step of fusing the 3D point-level features, the 3D voxel features, the 2D perspective features, and the 2D pixel features through a multimodal feature fusion module according to the mapping relationship to obtain the final fused features includes:

[0028] The 3D voxel features are used to find key 3D points near the center of each effective voxel at different resolutions. The 3D voxel features are aggregated with the 3D point-level features of the key 3D points to obtain a first aggregated feature. The first aggregated feature is then fused using the multimodal feature fusion module to obtain a first fused feature. The first fused feature is then used as the correction feature of the key 3D point.

[0029] Calculate the first and second mapping positions of the key 3D points in the initial perspective view and the two-dimensional image, respectively, based on the mapping relationship;

[0030] The 2D perspective features and 2D pixel features corresponding to the key 3D points are determined based on the first mapping position and the second mapping position.

[0031] The modified features, 2D perspective features, and 2D pixel features corresponding to the key 3D points are aggregated to obtain the second aggregated features. The features of the multimodal feature fusion module are then fused through a neural network layer to obtain the final fused features of the key 3D points.

[0032] Secondly, embodiments of the present invention provide a point cloud target detection device, comprising:

[0033] The acquisition module is used to acquire three-dimensional point cloud data through radar, acquire two-dimensional images through a camera, and obtain the mapping relationship between the three-dimensional point cloud data and the two-dimensional images.

[0034] The first extraction module is used to extract 3D point-level features from the three-dimensional point cloud data through the 3D sparse point feature extraction module.

[0035] The second extraction module is used to extract 3D voxel features from the three-dimensional point cloud data through the 3D sparse voxel feature extraction module.

[0036] The third extraction module is used to perform perspective projection on the three-dimensional point cloud data according to the radar perspective to obtain an initial perspective view, and to extract the 2D perspective features of the initial perspective view through the 2D feature extraction module.

[0037] The fourth extraction module is used to normalize the two-dimensional image to obtain normalized image data, and to extract 2D pixel features from the normalized image data through the 2D feature extraction module.

[0038] The processing module is used to fuse the 3D point-level features, the 3D voxel features, the 2D perspective features, and the 2D pixel features according to the mapping relationship through the multimodal feature fusion module to obtain the final fused features;

[0039] The prediction module is used to calculate the final fused features to obtain the three-dimensional information of the obstacle target.

[0040] Thirdly, embodiments of the present invention provide an electronic device, including a memory and a processor, wherein the memory is used to store a computer program, and the computer program executes the point cloud target detection method provided in the first aspect when the processor is running.

[0041] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program that, when run on a processor, executes the point cloud target detection method provided in the first aspect.

[0042] The point cloud target detection method, apparatus, electronic device, and storage medium provided in this application acquire three-dimensional point cloud data using radar and two-dimensional images using a camera to obtain the mapping relationship between the three-dimensional point cloud data and the two-dimensional images; extract 3D point-level features from the three-dimensional point cloud data using a 3D sparse point feature extraction module; extract 3D voxel features from the three-dimensional point cloud data using a 3D sparse voxel feature extraction module; perform perspective projection on the three-dimensional point cloud data according to the radar viewpoint to obtain an initial perspective view, and extract 2D perspective features from the initial perspective view using a 2D feature extraction module; normalize the two-dimensional images to obtain normalized image data, and extract 2D pixel features from the normalized image data using the 2D feature extraction module; fuse the 3D point-level features, the 3D voxel features, the 2D perspective features, and the 2D pixel features according to the mapping relationship using a multimodal feature fusion module to obtain a final fused feature; and calculate the final fused feature using a prediction module to obtain the three-dimensional information of the obstacle target. In this way, the existing target detection schemes address the problem of insufficient point cloud feature mining by making full use of the two different representation features of point cloud multimodal and image, deeply fusing 3D point-level features, 3D voxel features, 2D perspective features and 2D pixel features, and finally using rich fused features to predict the 3D boundary detection box of obstacle targets in the scene, thereby improving the detection accuracy of obstacle targets and achieving a more robust detection effect. Attached Figure Description

[0043] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope of protection of the present invention. In the various drawings, similar components are numbered similarly.

[0044] Figure 1 This paper illustrates a flowchart of a point cloud target detection method provided in an embodiment of this application.

[0045] Figure 2 A schematic diagram of the structure of the multimodal feature fusion model provided in an embodiment of this application is shown;

[0046] Figure 3 A schematic diagram of the point cloud target detection device provided in an embodiment of this application is shown.

[0047] Icons: 201-3D point cloud data; 202-calibration parameters; 203-2D image; 204-3D sparse point feature extraction module; 205-3D voxel point cloud; 206-initial perspective view; 207-normalization processing; 208-3D sparse voxel feature extraction module; 209-2D feature extraction module; 210-multimodal feature fusion module; 211-model prediction head; 212-final prediction result; 300-point cloud target detection device; 301-acquisition module; 302-first extraction module; 303-second extraction module; 304-third extraction module; 305-fourth extraction module; 306-processing module; 307-prediction module. Detailed Implementation

[0048] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0049] The components of the embodiments of the invention described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0050] In the following, the terms “comprising,” “having,” and their cognates, which may be used in various embodiments of the invention, are intended only to indicate a particular feature, number, step, operation, element, component, or combination thereof, and should not be construed as excluding, firstly, the presence of one or more other features, numbers, steps, operations, elements, components, or combinations thereof, or adding the possibility of one or more features, numbers, steps, operations, elements, components, or combinations thereof.

[0051] Furthermore, the terms "first," "second," and "third" are used only to distinguish descriptions and should not be interpreted as indicating or implying relative importance.

[0052] Unless otherwise specified, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which the various embodiments of the invention pertain. Terms (such as those defined in commonly used dictionaries) shall be interpreted as having the same meaning as in their contextual meaning in the relevant technical field and shall not be interpreted as having an idealized or overly formal meaning, unless clearly defined in the various embodiments of the invention.

[0053] Existing point cloud object detection schemes include the following: The first type is a point cloud object detection scheme based on raw point processing. This scheme processes the raw point cloud directly. First, it performs fixed-point sampling on the point cloud. Based on a point cloud scene with a certain number of points, it extracts features through methods such as hierarchical downsampling and grouping, upsampling and interpolation to obtain the semantic features of each point. This is used to segment the foreground and background in the point cloud scene and generate object detection boxes. This scheme retains the features of the raw point cloud to some extent, but it is quite sensitive to the total number of points, group radius, sampling ratio, and number of interpolation points. It is impossible to directly determine the impact of these parameters on detection through neural networks, and it is limited by collecting local features of the point cloud.

[0054] The second approach is a point cloud target detection scheme based on mesh voxelization. This method divides the original point cloud into several 3D meshes, calculates the coordinates and average reflectance of the points within each mesh, and uses these as the initial features for that mesh. Then, a 3D sparse convolutional backbone network extracts 3D voxel features, and these features are aggregated to ultimately complete target detection in the point cloud scene. This approach discards some information from the original points, and the size of the 3D mesh significantly impacts the network's detection performance. Furthermore, relying solely on voxelization features may result in insufficient robustness in localized areas.

[0055] The third type is a point cloud target detection scheme based on two-dimensional views. This scheme directly projects the 3D LiDAR onto a 2D plane according to specific mapping rules, such as projecting it onto a bird's-eye view or perspective view. It then uses a 2D image-based detection scheme to extract the features of the point cloud view. However, this type of method always loses feature information in a certain dimension, which is not very friendly to 3D target detection.

[0056] The aforementioned common methods process point cloud data according to a specific format, which can easily lead to insufficient representation of point cloud features. Furthermore, there are point cloud and 2D image fusion schemes based on single-modal processing. These schemes exhibit poor feature fusion results during the fusion of images and point clouds, failing to fundamentally increase feature richness to uncover the true data features of point clouds and images. To address these issues, this embodiment provides a point cloud target detection method, apparatus, electronic device, and storage medium, which will be described below.

[0057] Example 1

[0058] This disclosure provides a point cloud target detection method.

[0059] For details, see Figure 1 The point cloud target detection method includes the following steps:

[0060] Step S101: Collect three-dimensional point cloud data with radar and two-dimensional images with camera to obtain the mapping relationship between the three-dimensional point cloud data and the two-dimensional images.

[0061] In this embodiment, the movie target detection method can be applied to electronic devices equipped with radar and cameras. These electronic devices can be installed on intelligent vehicles, robots, or other intelligent devices to assist them in obstacle detection. Specifically, the radar can be a lidar system, and the camera can be a digital camera.

[0062] In this embodiment, the point cloud attributes of the 3D point cloud data include 3D coordinate information and reflectivity. The 3D point cloud data can be represented by N×4, where N represents the number of 3D points, 4 represents the number of channels, and the four channels represent the values ​​corresponding to the 3D coordinates (x, y, z) and reflectivity (intensity), respectively. The calibration parameters between the radar and the camera can be represented by a 4×4 rigid body rotation matrix, which is a transformation matrix from the radar coordinate system to the 2D image coordinate system. The image attributes of the 2D image include the RGB three-channel values ​​of the pixels. The 2D image can be represented by H×W×3, where H and W represent the length and width of the image, and 3 represents the RGB three channels.

[0063] In this embodiment, the three-dimensional point cloud data can be processed using the following steps S102, S103, and S104. It should be noted that steps S102, S103, and S104 can be performed according to... Figure 1 The execution order shown can be adjusted, but other orders are also possible, and simultaneous execution is also possible. Figure 1 The order shown is for illustrative purposes only and does not constitute a limitation on the execution order of steps S102, S103 and S104.

[0064] In one embodiment, the step S101 of obtaining the mapping relationship between the three-dimensional point cloud data and the two-dimensional image includes:

[0065] The mapping relationship between the pixels of the two-dimensional image and the three-dimensional points of the three-dimensional point cloud data is determined based on the calibration parameters between the radar and the camera.

[0066] Step S102: Extract 3D point-level features from the 3D point cloud data using the 3D sparse point feature extraction module.

[0067] In this embodiment, the 3D sparse point feature extraction module extracts 3D point-level features from the original 3D point cloud data acquired by radar, and the 3D point-level features can be used as the semantic features of the 3D points in the 3D point cloud data of each scene.

[0068] In one embodiment, step S102 includes:

[0069] The 3D sparse point feature extraction module performs fixed-point downsampling on the 3D point cloud data to obtain a fixed number of sparse point clouds, which serve as effective 3D point cloud data to characterize the current scene.

[0070] The features of sparse point clouds obtained through point cloud downsampling, point cloud grouping, feature extraction, and feature aggregation are used as 3D point-level features of the effective 3D point cloud data of the scene.

[0071] Please see Figure 2 The radar acquires 3D point cloud data 201 and the camera acquires 2D images 203, and determines the calibration parameters 202 between the radar and the camera. The 3D point cloud data 201 undergoes three aspects of processing. The first aspect of processing is described below. The first aspect of processing the 3D point cloud data 201 includes: performing fixed-point downsampling on the 3D point cloud data 201. During fixed-point downsampling, sparse points in the more distant areas of the region of interest in the 3D point cloud data 201 are retained, while some points in the more recent dense areas are discarded, to facilitate the subsequent sampling and feature calculation by the 3D sparse point feature extraction module.

[0072] For example, if the region of interest is 80m in front and behind and 50m in the left and right, the sparse point feature extraction module first performs fixed-point downsampling, discarding some points within the 40m front and behind and 30m left and right regions of the closer region, and retaining all points within the 40m front and behind and 20m left and right regions of the farther region, downsampling the overall point cloud to a fixed number of points. If there are 30,000 points in the region of interest, then 20,000 points can be selected as the effective 3D point cloud data of the scene. These effective 3D point clouds are downsampled sequentially to reduce the resolution of the point cloud, and the 3D semantic features of the downsampled point clouds at different resolutions are calculated. First, these effective 3D points are downsampled, and each point in the downsampled point cloud is used as the core point of a spherical group. Other points within a certain radius around each point are taken as a group. The point cloud features of each group are extracted through a designed neural network layer, and a certain number of group point features are aggregated to the core point of each spherical group through an aggregation operation, which serves as the 3D point-level feature of each 3D point after downsampling. Here, different downsampling points can be set according to the resolution of the point cloud, such as 10000, 5000, 2500, 1250, etc. Different spherical grouping radii can be set according to the number of downsampling points each time, such as 5cm, 10cm, 20cm, 30cm, etc. Different aggregation points can be set for each downsampling, such as 16, 32, etc., to calculate 3D point-level features at different resolutions.

[0073] Step S103: Extract 3D voxel features from the 3D point cloud data using the 3D sparse voxel feature extraction module.

[0074] In this embodiment, the 3D point cloud within the region of interest undergoes meshing preprocessing. Since most of the 3D point cloud scene contains blank areas, the preprocessing result only saves the 3D voxel point cloud containing actual points within the mesh, i.e., the effective 3D voxels. Each voxel point cloud corresponds to one 3D mesh in space. The 3D sparse voxel feature extraction module extracts 3D voxel features from the 3D voxel point cloud, which serve as local features for each 3D voxel point cloud and also as local region features for the points.

[0075] In one embodiment, step S103 includes:

[0076] Extract the region of interest (ROI) point cloud from the 3D point cloud data;

[0077] The point cloud of the region of interest is divided into voxels to obtain an effective three-dimensional sparse voxel point cloud. The attributes of the effective three-dimensional sparse voxel point cloud include initial features and three-dimensional index coordinates in space. The initial features are the average of three coordinate values ​​and reflectance values ​​of all points in each three-dimensional voxel.

[0078] The 3D sparse voxel feature extraction module extracts 3D voxel features from the effective 3D sparse voxel point cloud.

[0079] In one embodiment, the step of extracting 3D voxel features from the effective 3D sparse voxel point cloud using the 3D sparse voxel feature extraction module includes:

[0080] The mean value of the three-dimensional coordinate information of each three-dimensional point of the effective three-dimensional sparse voxel point cloud is determined, and the mean value of the reflectance of each three-dimensional point of the effective three-dimensional sparse voxel point cloud is determined. The mean value of the three-dimensional coordinate information and the mean value of the reflectance are used as the initial 3D voxel features of the effective three-dimensional sparse voxel point cloud, and the initial 3D voxel features are input into the 3D sparse voxel feature extraction module.

[0081] The 3D sparse voxel feature extraction module determines the feature output coordinate information based on the 3D coordinate information of the effective 3D sparse voxel point cloud, and performs convolution operation on the initialized voxel features based on the 3D coordinate information of the effective 3D sparse voxel point cloud and the feature output coordinate information to obtain the 3D voxel features.

[0082] It should be noted that the effective coordinate position of the 3D sparse voxel point cloud refers to the 3D coordinate index of the grid. For example, if the point cloud data range is x: 0-40m, y: 0-40m, z: 0-2m, and the grid size is set to (0.1m, 0.1m, 0.1m), then it will be divided into 400*400*20 voxels. Points within a voxel are called effective voxels. The characteristic of an effective voxel is that the actual xyz coordinates and reflectance of the points inside it are added together and the average value is calculated. The coordinate position of an effective voxel is represented by a 3D spatial index. For example, the voxel containing the point (0.05, 0.05, 0.05) has coordinates of (0, 0, 0).

[0083] Please refer to it again. Figure 2 The following is combined with Figure 2 The second aspect of processing the 3D point cloud data 201 is described below. This second aspect of processing includes: 3D mesh voxelization of the 3D point cloud data 201; setting a fixed size for the 3D mesh; dividing the 3D point cloud within the area of ​​interest into H0×W0×L0 voxels, where H0, W0, and L0 represent the length, width, and height of the voxel, respectively; and H0×W0×L0 representing the number of voxels, thus obtaining multiple 3D voxel point clouds 205. The average of the four channels of each 3D point within the voxel is calculated as the initial 3D voxel feature for that voxel. The initial 3D voxel feature is represented as M×4, where M represents the number of effective voxels and 4 represents the number of initial feature channels for the voxel. The corresponding coordinate position is M×3, where M represents the number of effective voxels and 3 represents the number of integer coordinate channels for the voxel. The 3D voxel features and voxel coordinates are input into the 3D sparse voxel feature extraction module 208. Based on the coordinates of the effective voxels, the possible output feature coordinate positions are calculated. Based on the 3D coordinate positions of the effective input voxels, the size of the convolution kernel, the stride of the convolution kernel, and other attribute information, the 3D coordinate positions of the corresponding effective output voxel features are calculated. Convolution operations are performed on the features according to the coordinate mapping, ultimately generating 3D voxel features. The 3D voxel features are represented as N×C1, where N represents the number of effective outputs, and C1 represents the number of output channels. Different strides are set to calculate point cloud 3D voxel features at multiple scales. C1 is the output feature, typically set to 256 or 128, but can be adjusted empirically.

[0084] Step S104: Perform perspective projection on the three-dimensional point cloud data according to the radar perspective to obtain an initial perspective view, and extract the 2D perspective features of the initial perspective view through the 2D feature extraction module.

[0085] In this embodiment, the original three-dimensional point cloud data acquired by the radar is projected from the radar perspective to obtain an initial perspective view. The initial features of the initial perspective view are the distance, angle and line bundle information of the points. The point cloud features of the initial perspective view are extracted based on the 2D feature extraction module as local structured features of the three-dimensional point cloud data under the radar perspective.

[0086] In one embodiment, step S104, which involves performing perspective projection on the three-dimensional point cloud data according to a radar view to obtain an initial perspective view, includes:

[0087] Calculate the Euclidean distance and elevation angle between each three-dimensional point in the three-dimensional point cloud data and the radar;

[0088] Based on the Euclidean distance, pitch angle, wiring harness, and 3D coordinate information of each 3D point, map each 3D point onto a preset 2D perspective view to obtain the initial perspective view.

[0089] Please refer to it again. Figure 2 The following is combined with Figure 2 The third aspect of processing the 3D point cloud data 201 is described below. This third aspect of processing includes: calculating the Euclidean distance between each 3D point in the 3D point cloud data and the radar; calculating the elevation angle between each 3D point in the 3D point cloud data and the radar; and mapping the point cloud onto a preset 2D perspective view of a fixed size based on its location within the wiring harness, elevation angle, Euclidean distance, and 3D coordinates, thereby generating an initial perspective view 206. This initial perspective view is represented as H1×W1×5, where H1 and W1 represent the length and width of the image, respectively, and 5 represents the number of feature channels. ， Each channel sequentially represents the radar observation range, x-coordinate, y-coordinate, z-coordinate, and radar observation elevation angle. The 2D feature extraction module 209 extracts depth features from the initial perspective view 206 to obtain the 2D perspective features of the point cloud. The 2D perspective features are represented as H1×W1×C2, where H1 and W1 represent the image width and height, and C2 represents the output feature channel. ， It can generally be set to 512 or 256.

[0090] Step S105: Normalize the two-dimensional image to obtain normalized image data, and extract 2D pixel features from the normalized image data using the 2D feature extraction module.

[0091] In this embodiment, the two-dimensional image is first normalized, and the pixel-level features are extracted by the 2D feature extraction module as two-dimensional image features. The aforementioned four features—semantic features, local region features, local structured features, and two-dimensional image features—together reflect the semantic information of the three-dimensional point cloud data in the current scene.

[0092] Please refer to it again. Figure 2The following is combined with Figure 2 The normalization process for the two-dimensional image is explained. Specifically, the mean pixel value and scaling factor of the two-dimensional image 203 are calculated. The normalization process 207 specifically includes: subtracting the mean pixel value from the pixel value of the two-dimensional image 203 to obtain the difference, and dividing the difference by the scaling factor to obtain the quotient. The image size of the normalized image data is H2×W2×3, where H2 and W2 represent the length and width of the image, respectively, and 3 represents the number of RGB channels. The 2D feature extraction module 209 performs depth feature extraction on the normalized image data to obtain the 2D pixel features of the normalized image data. The size of the 2D pixel features is H2×W2×C3, where H2 and W2 represent the length and width of the image, respectively, and C3 represents the number of output feature channels, which can generally be set to 512 or 256.

[0093] Step S106: The multimodal feature fusion module fuses the 3D point-level features, the 3D voxel features, the 2D perspective features, and the 2D pixel features according to the mapping relationship to obtain the final fused features.

[0094] In this embodiment, the multimodal feature fusion module receives 3D point-level features, 3D voxel features, 2D perspective features, and 2D pixel features. These features can be fused at different scales. The fusion principle is to determine the mapping relationship between pixels and point clouds through the calibration parameters of the camera and radar. First, the 3D voxel features are grouped and matched with the 3D points of the original 3D point cloud data. The 3D voxel features are then fused to the matching 3D points in the original 3D point cloud data. Then, the fusion of 3D point-level features, 2D perspective features, and 2D pixel features is realized to obtain the final fused feature. The fused feature map is used as the refined feature of the 3D point cloud data.

[0095] In one embodiment, step S106 includes:

[0096] The 3D voxel features are used to find key 3D points near the center of each effective voxel at different resolutions. The 3D voxel features are aggregated with the 3D point-level features of the key 3D points to obtain a first aggregated feature. The first aggregated feature is then fused using the multimodal feature fusion module to obtain a first fused feature. The first fused feature is then used as the correction feature of the key 3D point.

[0097] Calculate the first and second mapping positions of the key 3D points in the initial perspective view and the two-dimensional image, respectively, based on the mapping relationship;

[0098] The 2D perspective features and 2D pixel features corresponding to the key 3D points are determined based on the first mapping position and the second mapping position.

[0099] The modified features, 2D perspective features, and 2D pixel features corresponding to the key 3D points are aggregated to obtain the second aggregated features. The features of the multimodal feature fusion module are then fused through a neural network layer to obtain the final fused features of the key 3D points.

[0100] Please refer to it again. Figure 2 The following is combined with Figure 2 The processing flow of the multimodal feature fusion module 210 is described below. Specifically, the multimodal feature fusion module 210 receives 3D point-level features, 3D voxel features, 2D perspective features, and 2D pixel features from the 3D sparse point feature extraction module 204, the 3D sparse voxel feature extraction module 208, and the 2D feature extraction module 209, respectively. The multimodal feature fusion module 210 first searches for key 3D points near the center position of each effective voxel according to different resolutions of the 3D voxel features. It then aggregates the 3D voxel features with the 3D point-level features of the key 3D points to obtain the first aggregated feature. The neural network layer of the multimodal feature fusion module 210 performs feature fusion on the first aggregated feature to obtain the first fused feature, which is used as the local feature of the key 3D point.

[0101] Furthermore, the first and second mapping positions of key 3D points in the initial perspective view and the 2D image are calculated respectively. Based on the first and second mapping positions, the corresponding 2D perspective features and 2D pixel features of the key 3D points are determined. The local features, 2D perspective features, and 2D pixel features of the key 3D points are aggregated to obtain a second aggregated feature. The second aggregated feature is then fused through the neural network layer of the multimodal feature fusion module 210 to obtain the final fused feature of the key 3D points. The final fused feature of the key 3D points is represented as a feature map of size K×C4, where K represents the number of effective key 3D points and C4 represents the number of fused feature channels. It should be noted that in this embodiment, data aggregation can be understood as a simple concatenation operation, while feature fusion refers to deep feature fusion through conv / bn / relu layers; these are two different processing procedures.

[0102] Step S107: The prediction module calculates the final fused features to obtain the three-dimensional information of the obstacle target.

[0103] In one embodiment, the multimodal feature fusion module outputs the fused features to the prediction module, which predicts the three-dimensional information of the obstacle, and finally the network outputs the three-dimensional information of the detected obstacle target.

[0104] Please refer to it again. Figure 2 The following is combined with Figure 2The processing flow of the prediction module is described below. Specifically, the prediction module includes a model prediction head 211, which mainly predicts the information of obstacle targets. The model prediction head 211 includes category prediction, 3D bounding box information prediction, and orientation prediction. The prediction results and preset prior boxes (anchors) are post-processed and calculated. Finally, the final prediction result 212 is obtained through non-maximum suppression (NMS) operation. The final prediction result 212 includes the three-dimensional information of the obstacle targets.

[0105] The target detection method disclosed in this embodiment acquires three-dimensional point cloud data using radar and two-dimensional images using a camera, obtaining the mapping relationship between the three-dimensional point cloud data and the two-dimensional images; extracts 3D point-level features from the three-dimensional point cloud data using a 3D sparse point feature extraction module; extracts 3D voxel features from the three-dimensional point cloud data using a 3D sparse voxel feature extraction module; performs perspective projection on the three-dimensional point cloud data according to the radar viewpoint to obtain an initial perspective view, and extracts 2D perspective features from the initial perspective view using a 2D feature extraction module; normalizes the two-dimensional images to obtain normalized image data, and extracts 2D pixel features from the normalized image data using the 2D feature extraction module; fuses the 3D point-level features, the 3D voxel features, the 2D perspective features, and the 2D pixel features according to the mapping relationship using a multimodal feature fusion module to obtain a final fused feature; and calculates the final fused feature using a prediction module to obtain the three-dimensional information of the obstacle target. In this way, the existing target detection schemes address the problem of insufficient point cloud feature mining by making full use of the two different representation features of point cloud multimodal and image, deeply fusing 3D point-level features, 3D voxel features, 2D perspective features and 2D pixel features, and finally using rich fused features to predict the 3D boundary detection box of obstacle targets in the scene, thereby improving the detection accuracy of obstacle targets and achieving a more robust detection effect.

[0106] Example 2

[0107] This disclosure also provides a point cloud target detection device.

[0108] For details, please refer to Figure 3 The point cloud target detection device 300 includes:

[0109] The acquisition module 301 is used to acquire three-dimensional point cloud data through radar, acquire two-dimensional images through camera, and obtain the mapping relationship between the three-dimensional point cloud data and the two-dimensional images.

[0110] The first extraction module 302 is used to extract 3D point-level features from the three-dimensional point cloud data through the 3D sparse point feature extraction module.

[0111] The second extraction module 303 is used to extract 3D voxel features from the three-dimensional point cloud data through the 3D sparse voxel feature extraction module.

[0112] The third extraction module 304 is used to perform perspective projection on the three-dimensional point cloud data according to the radar perspective to obtain an initial perspective view, and to extract the 2D perspective features of the initial perspective view through the 2D feature extraction module.

[0113] The fourth extraction module 305 is used to normalize the two-dimensional image to obtain normalized image data, and to extract 2D pixel features from the normalized image data through the 2D feature extraction module.

[0114] Processing module 306 is used to fuse the 3D point-level features, the 3D voxel features, the 2D perspective features, and the 2D pixel features according to the mapping relationship through a multimodal feature fusion module to obtain the final fused features;

[0115] The prediction module 307 is used to calculate the final fused features through the prediction module to obtain the three-dimensional information of the obstacle target.

[0116] In one embodiment, the acquisition module 301 is further configured to determine the mapping relationship between the pixels of the two-dimensional image and the three-dimensional points of the three-dimensional point cloud data based on the calibration parameters between the radar and the camera.

[0117] In one embodiment, the first extraction module 302 is further configured to perform fixed-point downsampling processing on the three-dimensional point cloud data through the 3D sparse point feature extraction module to obtain a fixed number of sparse point clouds as effective three-dimensional point cloud data characterizing the current scene.

[0118] The features of sparse point clouds obtained through point cloud downsampling, point cloud grouping, feature extraction, and feature aggregation are used as 3D point-level features of the effective 3D point cloud data of the scene.

[0119] In one embodiment, the second extraction module 303 is further configured to extract the region of interest point cloud from the three-dimensional point cloud data;

[0120] The point cloud of the region of interest is divided into voxels to obtain an effective three-dimensional sparse voxel point cloud. The attributes of the effective three-dimensional sparse voxel point cloud include initial features and three-dimensional index coordinates in space. The initial features are the average of three coordinate values ​​and reflectance values ​​of all points in each three-dimensional voxel.

[0121] The 3D sparse voxel feature extraction module extracts 3D voxel features from the effective 3D sparse voxel point cloud.

[0122] In one embodiment, the second extraction module 303 is further configured to determine the mean value of the three-dimensional coordinate information of each three-dimensional point of the effective three-dimensional sparse voxel point cloud, and determine the mean value of the reflectance of each three-dimensional point of the effective three-dimensional sparse voxel point cloud, use the mean value of the three-dimensional coordinate information and the mean value of the reflectance as the initial 3D voxel features of the effective three-dimensional sparse voxel point cloud, and input the initial 3D voxel features into the 3D sparse voxel feature extraction module;

[0123] The 3D sparse voxel feature extraction module determines the feature output coordinate information based on the 3D coordinate information of the effective 3D sparse voxel point cloud, and performs convolution operation on the initialized voxel features based on the 3D coordinate information of the effective 3D sparse voxel point cloud and the feature output coordinate information to obtain the 3D voxel features.

[0124] In one embodiment, the third extraction module 304 is further used to calculate the Euclidean distance and pitch angle between each three-dimensional point of the three-dimensional point cloud data and the radar;

[0125] Based on the Euclidean distance, pitch angle, wiring harness, and 3D coordinate information of each 3D point, map each 3D point onto a preset 2D perspective view to obtain the initial perspective view.

[0126] In one embodiment, the fourth extraction module 305 is used to find key 3D points near the center position of each effective voxel according to different resolutions of the 3D voxel features, aggregate the 3D voxel features and the 3D point-level features of the key 3D points to obtain a first aggregated feature, perform feature fusion on the first aggregated feature through the multimodal feature fusion module to obtain a first fused feature, and use the first fused feature as the correction feature of the key 3D point.

[0127] Calculate the first and second mapping positions of the key 3D points in the initial perspective view and the two-dimensional image, respectively, based on the mapping relationship;

[0128] The 2D perspective features and 2D pixel features corresponding to the key 3D points are determined based on the first mapping position and the second mapping position.

[0129] The modified features, 2D perspective features, and 2D pixel features corresponding to the key 3D points are aggregated to obtain the second aggregated features. The features of the multimodal feature fusion module are then fused through a neural network layer to obtain the final fused features of the key 3D points.

[0130] The point cloud target detection device provided in this embodiment acquires three-dimensional point cloud data through radar and two-dimensional images through a camera, obtaining the mapping relationship between the three-dimensional point cloud data and the two-dimensional images; it extracts 3D point-level features from the three-dimensional point cloud data through a 3D sparse point feature extraction module; it extracts 3D voxel features from the three-dimensional point cloud data through a 3D sparse voxel feature extraction module; it performs perspective projection on the three-dimensional point cloud data according to the radar viewpoint to obtain an initial perspective view, and extracts 2D perspective features from the initial perspective view through a 2D feature extraction module; it normalizes the two-dimensional images to obtain normalized image data, and extracts 2D pixel features from the normalized image data through the 2D feature extraction module; it fuses the 3D point-level features, the 3D voxel features, the 2D perspective features, and the 2D pixel features according to the mapping relationship through a multimodal feature fusion module to obtain a final fused feature; and it calculates the final fused feature through a prediction module to obtain the three-dimensional information of the obstacle target. In this way, the existing target detection schemes address the problem of insufficient point cloud feature mining by making full use of the two different representation features of point cloud multimodal and image, deeply fusing 3D point-level features, 3D voxel features, 2D perspective features and 2D pixel features, and finally using rich fused features to predict the 3D boundary detection box of obstacle targets in the scene, thereby improving the detection accuracy of obstacle targets and achieving a more robust detection effect.

[0131] Example 3

[0132] This disclosure also provides an electronic device, including a memory and a processor, wherein the memory is used to store a computer program, and the computer program executes the point cloud target detection method provided in Embodiment 1 when the processor is running.

[0133] The electronic device provided in this disclosure can implement the steps of the point cloud target detection method provided in Embodiment 1. To avoid repetition, it will not be described in detail here.

[0134] Example 4

[0135] This disclosure also provides a computer-readable storage medium storing a computer program that, when run on a processor, executes the point cloud target detection method provided in Embodiment 1.

[0136] The computer-readable storage medium provided in this disclosure embodiment can implement the steps of the point cloud target detection method provided in Embodiment 1. To avoid repetition, it will not be described in detail here.

[0137] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative; for example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that, as an alternative implementation, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0138] In addition, the functional modules or units in the various embodiments of the present invention can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0139] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device (which may be a smartphone, personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0140] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A point cloud object detection method, characterized in that, The method comprises the following steps: acquiring a mapping relationship between the three-dimensional point cloud data and the two-dimensional image by collecting three-dimensional point cloud data through a radar and collecting a two-dimensional image through a camera; extracting 3D point-level features of the three-dimensional point cloud data through a 3D sparse point feature extraction module; extracting 3D voxel features of the three-dimensional point cloud data through a 3D sparse voxel feature extraction module; performing perspective projection on the three-dimensional point cloud data according to a radar perspective to obtain an initial perspective view, and extracting 2D perspective features of the initial perspective view through a 2D feature extraction module; performing normalization processing on the two-dimensional image to obtain normalized image data, and extracting 2D pixel features of the normalized image data through the 2D feature extraction module; fusing the 3D point-level features, the 3D voxel features, the 2D perspective features and the 2D pixel features according to the mapping relationship through a multi-modal feature fusion module to obtain final fusion features; including, searching for key 3D points near each effective voxel center position according to different resolutions of the 3D voxel features, aggregating the 3D voxel features and the 3D point-level features of the key 3D points to obtain first aggregation features, performing feature fusion on the first aggregation features through the multi-modal feature fusion module to obtain first fusion features, and taking the first fusion features as modified features of the key 3D points; calculating first mapping positions and second mapping positions of the key 3D points in the initial perspective view and the two-dimensional image respectively according to the mapping relationship; determining 2D perspective features and 2D pixel features corresponding to the key 3D points according to the first mapping positions and the second mapping positions; aggregating the modified features, the 2D perspective features and the 2D pixel features corresponding to the key 3D points to obtain second aggregation features, and performing feature fusion on the second aggregation features through a neural network layer of the multi-modal feature fusion module to obtain final fusion features of the key 3D points; calculating the final fusion features through a prediction module to obtain three-dimensional information of an obstacle target.

2. The method of claim 1, wherein, The step of acquiring the mapping relationship between the three-dimensional point cloud data and the two-dimensional image comprises: determining the mapping relationship between pixel points of the two-dimensional image and three-dimensional points of the three-dimensional point cloud data according to calibration parameters between the radar and the camera.

3. The method of claim 1, wherein, The step of extracting 3D point-level features of the three-dimensional point cloud data through the 3D sparse point feature extraction module comprises: performing fixed-point downsampling processing on the three-dimensional point cloud data through the 3D sparse point feature extraction module to obtain a fixed number of sparse point clouds as effective three-dimensional point cloud data representing a current scene; obtaining features of the sparse point clouds through point cloud downsampling, point cloud grouping, feature extraction and feature aggregation as the 3D point-level features of the effective three-dimensional point cloud data of the scene.

4. The method of claim 1, wherein, The step of extracting 3D voxel features of the three-dimensional point cloud data through the 3D sparse voxel feature extraction module comprises: extracting region-of-interest point clouds from the three-dimensional point cloud data; The voxel division processing is performed on the region-of-interest point cloud to obtain an effective three-dimensional sparse voxel point cloud, and attributes of the effective three-dimensional sparse voxel point cloud include initial features and three-dimensional index coordinate positions of a space, and the initial features are average values of three coordinate values and reflectivity values of all points in each three-dimensional voxel; The 3D voxel features are extracted from the effective three-dimensional sparse voxel point cloud by the 3D sparse voxel feature extraction module.

5. The method of claim 4, wherein, The step of extracting the 3D voxel features from the effective three-dimensional sparse voxel point cloud by the 3D sparse voxel feature extraction module includes: The average values of three-dimensional coordinate information of each three-dimensional point of the effective three-dimensional sparse voxel point cloud are determined, and the average values of reflectivity of each three-dimensional point of the effective three-dimensional sparse voxel point cloud are determined, the average values of the three-dimensional coordinate information and the average values of the reflectivity are taken as initial 3D voxel features of the effective three-dimensional sparse voxel point cloud, and the initial 3D voxel features are input into the 3D sparse voxel feature extraction module; The feature output coordinate information is determined according to the three-dimensional coordinate information of the effective three-dimensional sparse voxel point cloud by the 3D sparse voxel feature extraction module, the convolution operation is performed on the initial 3D voxel features according to the three-dimensional coordinate information of the effective three-dimensional sparse voxel point cloud and the feature output coordinate information, and the 3D voxel features are obtained.

6. The method of claim 1, wherein, The step of performing perspective projection on the three-dimensional point cloud data according to a radar perspective to obtain an initial perspective view includes: The Euclidean distances and pitch angles between each three-dimensional point of the three-dimensional point cloud data and the radar are calculated; Each three-dimensional point is mapped to a preset 2D perspective view according to the corresponding Euclidean distance, pitch angle, line beam and three-dimensional coordinate information of each three-dimensional point, and the initial perspective view is obtained.

7. A point cloud object detection apparatus, characterized by comprising: The device includes: An acquisition module is configured to collect three-dimensional point cloud data by a radar and collect two-dimensional images by a camera, and obtain a mapping relationship between the three-dimensional point cloud data and the two-dimensional images; A first extraction module is configured to extract 3D point-level features from the three-dimensional point cloud data by a 3D sparse point feature extraction module; A second extraction module is configured to extract 3D voxel features from the three-dimensional point cloud data by a 3D sparse voxel feature extraction module; A third extraction module is configured to perform perspective projection on the three-dimensional point cloud data according to a radar perspective to obtain an initial perspective view, and extract 2D perspective features of the initial perspective view by a 2D feature extraction module; A fourth extraction module is configured to perform normalization processing on the two-dimensional images to obtain normalized image data, and extract 2D pixel features from the normalized image data by the 2D feature extraction module. The processing module is configured to fuse the 3D point-level feature, the 3D voxel feature, the 2D perspective feature and the 2D pixel feature according to the mapping relationship by the multi-modal feature fusion module to obtain a final fusion feature; including, searching for key 3D points near each effective voxel center position according to different resolutions by the 3D voxel feature, aggregating the 3D voxel feature and the 3D point-level feature of the key 3D points to obtain a first aggregation feature, performing feature fusion on the first aggregation feature by the multi-modal feature fusion module to obtain a first fusion feature, taking the first fusion feature as a correction feature of the key 3D points; calculating a first mapping position and a second mapping position of the key 3D points in the initial perspective view and the two-dimensional image respectively according to the mapping relationship; determining 2D perspective features and 2D pixel features corresponding to the key 3D points according to the first mapping position and the second mapping position; aggregating the correction feature, the 2D perspective features and the 2D pixel features corresponding to the key 3D points to obtain a second aggregation feature, and performing feature fusion on the second aggregation feature by a feature neural network layer of the multi-modal feature fusion module to obtain a final fusion feature of the key 3D points; The prediction module is configured to calculate the final fusion feature by the prediction module to obtain three-dimensional information of the obstacle target.

8. An electronic device, comprising: The computer program is stored in the memory and executed on the processor to perform the point cloud target detection method in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The computer program is stored in the memory and executed on the processor to perform the point cloud target detection method in any one of claims 1 to 7.

Images