Point cloud image fusion target detection method based on multi-scale voxel feature aggregation

Abstract

The application discloses a point cloud image fusion target detection method based on multi-scale voxel feature aggregation, comprising the following steps: extracting point cloud voxel features and image features; fusing point cloud voxel features and image features; and aggregating multi-scale voxel features. The center of each non-empty voxel is taken as a key point, which is projected onto an image plane through an external parameter matrix of a camera-laser radar and a camera internal parameter, and rich texture and color features are extracted from corresponding image features by using a bilinear interpolation method as supplementary information of the point cloud voxel features, so that the point cloud voxel features and the image features are globally fused, the semantic information of the point cloud voxel features is enhanced, the inherent limitation of the laser radar sensor is made up, and the accuracy of a detection result is improved; and the multi-scale voxels generated in the voxel coding stage are used to realize cross-scale extraction of sparse features, enhance the representation ability of the model to multi-scale targets, and further improve the detection performance and positioning ability of the model.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving, and in particular to a point cloud image fusion target detection method based on multi-scale voxel feature aggregation. Background Technology

[0002] Environmental perception is a primary capability for autonomous vehicles. Object detection, which acquires information such as the category, location, and orientation of other objects in the surrounding environment, is an essential component of environmental perception. LiDAR provides accurate depth information and rich geometric feature information, making it the mainstream data source for object detection. However, due to the inherent limitations of LiDAR sensors, point clouds are typically sparse, failing to provide sufficient context to distinguish distant areas, resulting in poor performance and limiting the detection of key targets. Furthermore, it is difficult to distinguish objects with similar appearances based solely on geometric shape. Conversely, cameras can provide high-resolution images containing rich texture, color, and shape information, but perform poorly in low-light and adverse weather conditions. The complementary information from LiDAR and cameras, and the effective fusion of point cloud and image features, can improve the detection of key targets in complex scenes, representing a crucial approach to enhancing detection performance.

[0003] Due to the many uncertainties in real-world driving scenarios, small targets in road scenes, such as pedestrians, cyclists, and distant vehicles, are often difficult to detect due to a lack of detail. This affects the detection accuracy of target detection methods, easily leading to vehicle collisions and seriously threatening driving safety. Currently, some solutions consider both point clouds and images for detection, such as a multi-source heterogeneous sensor anti-interference fusion perception system (patent application number: 202511405579.4). This system comprehensively considers information from LiDAR, cameras, and millimeter-wave radar, and uses an adaptive fusion module to dynamically adjust the weights of each modality based on weather conditions. This effectively improves the perception accuracy and reliability of vehicles in complex environments such as rain and fog, ensuring driving safety. However, it overlooks the problem that small targets in road scenes are often difficult to detect due to a lack of detail. Summary of the Invention

[0004] To address the aforementioned problems in existing technologies, this invention discloses a point cloud image fusion target detection method based on multi-scale voxel feature aggregation, which can significantly improve the detection accuracy of small targets in complex scenes and enhance the reliability of autonomous vehicles' perception of their surrounding environment.

[0005] To achieve the above objectives, the basic idea of ​​this invention is as follows: First, the point cloud is divided into voxels according to a preset grid size. A sparse convolutional network is used to extract voxel features, and a Swin-Tiny network and Feature Pyramid Networks (FPN) are used to extract image features. Second, in the voxel feature encoding stage, the center of each non-empty voxel is used as a key point. The key point is projected onto the image plane using the extrinsic matrix of the camera-LiDAR and the intrinsic parameters of the camera. Bilinear interpolation is used to extract texture and color information from the corresponding positions of the image features to supplement the deficiencies of the voxel features. Then, based on the multi-scale voxels in the voxel encoding stage, a multi-scale voxel feature aggregation network is constructed to fuse voxel features of different resolutions, enhancing the model's ability to express features of multi-scale targets. Finally, the bird's-eye view (BEV) features output by the multi-scale voxel feature aggregation network are input into the detection head to achieve accurate detection and localization of cars, pedestrians, and cyclists.

[0006] The technical solution of this invention is as follows: A point cloud image fusion target detection method based on multi-scale voxel feature aggregation, comprising the following steps: A. Extracting point cloud voxel features and image features A1. Collect initial point cloud data of the surrounding environment. ,in, This indicates the number of points in the point cloud of the current frame. Represents the initial features of the point cloud. The first The points are in the lidar coordinate system. x axis, y axis, z Coordinates on the axis For the first The reflectance at each point.

[0007] A2. Define the valid area for point cloud data as follows: , They are the starting ones x axis, y axis, z The coordinates of the axis, They are terminated respectively x axis, y axis, z The coordinates of the axes are obtained through the effective area of ​​the point cloud data. Acquire point cloud data within the effective area :

[0008] In the formula, To effectively divide the network into regions, For the initial point cloud data, This refers to the valid area of ​​the point cloud data.

[0009] A3. Set the size of the voxel mesh. ,in, For voxel mesh in x axis, y axis, z The length on the axis, for point cloud data within the effective area. Divide into ,in, This represents the characteristics of the points contained in each voxel. Represents the coordinates of each voxel. The formula for representing the number of valid points in each voxel is as follows:

[0010] In the formula, For voxel generation networks, The maximum number of sampling points for each voxel is defined. When the actual number of points exceeds the threshold, a threshold point cloud is retained through random sampling. When the number of points is less than the threshold, zero vector padding is used. This is the maximum number of voxels per frame. If the limit is exceeded, voxels exceeding the limit will be discarded by default.

[0011] A4. At this point, voxel characteristics Including each voxel The characteristics of each point are determined based on the number of valid points in each voxel. By targeting effective points within voxels x axis, y axis, z The effective point fusion feature within each voxel is obtained by averaging along the axis and reflection intensity dimensions. :

[0012] In the formula, To target effective points within voxels x axis, y axis, z Calculate the mean value along the axis and reflection intensity dimensions. In each voxel Features of each point The number of valid points in each voxel.

[0013] A5. Using a 3D sparse convolutional network as the backbone network for point cloud features to extract sparse tensors of voxel features. The formula is as follows:

[0014] In the formula, It is a three-dimensional sparse convolutional network. For sparse tensor generation networks, Features fused at effective points within each voxel. The coordinates of each voxel, These are the weights of a 3D sparse convolutional network.

[0015] A6. Acquire initial image data of the surrounding environment. The Swin-Tiny network was used as the image feature extraction network to extract multi-stage image features. A feature pyramid network is used to fuse multi-stage image features. Obtaining fused image features :

[0016]

[0017] In the formula, For the Swin-Tiny network, For initial image data, The weights of the Swin-Tiny network For feature pyramid network, The weights are those of the feature pyramid network.

[0018] B. Integrating point cloud voxel features and image features B1. For sparse tensors For each non-empty voxel, take the coordinates of its three-dimensional center point. As a key point, among them, Indicates the number of non-empty voxels. Indicates the first The coordinates of the three-dimensional center point of a non-empty voxel in the lidar coordinate system The first The central point is x axis, y axis, z Coordinates on the axis. Using pre-calibrated extrinsic and intrinsic parameters of the laser-laser and camera, the coordinates of the voxel center point are... Projecting the image pixel coordinates from the lidar coordinate system to the image pixel coordinate system yields the projected image pixel coordinates. ,in, , The first The projection points are in the image pixel coordinate system x axis, y The coordinates on the axis are given by the following formula:

[0019] In the formula, Representing the The depth of each projection point K It is the camera intrinsic parameter matrix. It is the extrinsic parameter matrix of the lidar to the camera. , , The first Individual pixel center point in lidar coordinate system x axis, y axis, z Coordinates on the axis.

[0020] B2. In image features Above, for the pixel coordinates of the projected image The corresponding image feature vector is obtained using bilinear interpolation. :

[0021] In the formula, For bilinear interpolation, For image features, These are the pixel coordinates of the projected image.

[0022] B3. Extract the image feature vector With sparse tensors voxel feature vectors The features are then concatenated and then subjected to dimensionality reduction and feature integration through a fully connected layer to obtain the enhanced voxel features. :

[0023] In the formula, It is a fully connected layer. For feature concatenation function, For sparse tensors voxel feature vectors, This is the extracted image feature vector.

[0024] B4. Based on the enhanced voxel characteristics and sparse tensors voxel coordinates New sparse tensors are generated, and three-stage fusion features are extracted using three 3D sparse convolutional networks. , , ,in, The fused features are extracted through a 3D sparse convolutional network. The fused features are extracted from two 3D sparse convolutional networks. The fused features extracted by three 3D sparse convolutional networks are expressed in the following formula:

[0025]

[0026]

[0027] In the formula, It is a three-dimensional sparse convolutional network. For sparse tensor generation networks, For enhanced voxel features, For sparse tensors voxel coordinates, These are the weights of a 3D sparse convolutional network.

[0028] C. Aggregated multiscale voxel features C1. Features are integrated through a three-stage flattening process in the height direction. ,in, , , This indicates that the three-stage fusion features extracted by the above three 3D sparse convolutional networks are used to obtain multi-scale BEV features. The formula is as follows:

[0029] In the formula, It is a convolutional neural network. For a network that is flattened in the height direction, The above refers to the three-stage fusion features extracted by three 3D sparse convolutional networks. The weights are the weights of the convolutional neural network, and the BEV features are the bird's-eye view features.

[0030] C2. At each scale, BEV features obtained by upsampling voxels at the current scale are fused with smaller BEV features from the previous scale, as shown in the following formula:

[0031]

[0032] In the formula, The BEV features are the result of multi-scale voxel feature aggregation. , The multi-scale BEV features are obtained by voxel feature compression. It is a convolutional neural network. For upsampling networks, These are the weights of the convolutional neural network.

[0033] C3. BEV features after aggregating multi-scale voxel features Input is fed into the neck and head to decode and output the target's location and category information.

[0034] Compared with the prior art, the present invention has the following beneficial effects: 1. In the voxel feature encoding stage, this invention uses the center of each non-empty voxel as a key point. Through the extrinsic parameter matrix of the camera-LiDAR and the intrinsic parameters of the camera, it projects the voxel onto the image plane. Then, it uses bilinear interpolation to extract rich texture and color features from the corresponding image features as supplementary information for the point cloud voxel features. The point cloud voxel features and image features are globally fused to enhance the semantic information of the point cloud voxel features, make up for the inherent limitations of the LiDAR sensor, and improve the accuracy of the detection results.

[0035] 2. This invention utilizes multi-scale voxels generated during the voxel encoding stage to achieve cross-scale extraction of sparse features. In each scale, BEV features are obtained by compressing voxels of the current scale and fusing features of smaller voxels from the previous scale. This achieves feature aggregation while maintaining the spatial structure of the previous scale, enhancing the model's ability to represent multi-scale targets and thereby improving the model's detection and localization capabilities. Attached Figure Description

[0036] This invention has a total of appendices Figure 3 Zhang, of which: Figure 1 This is a flowchart of the present invention.

[0037] Figure 2 This is a schematic diagram of the framework of the present invention.

[0038] Figure 3 It is a diagram of the multi-scale voxel feature aggregation network structure. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. Figure 1 The flowchart shown below illustrates a point cloud image fusion target detection method based on multi-scale voxel feature aggregation, which includes the following steps: A. Extracting point cloud voxel features and image features A1. Collect initial point cloud data of the surrounding environment. ,in, This indicates the number of points in the point cloud of the current frame. Represents the initial features of the point cloud. The first The points are in the lidar coordinate system. x axis, y axis,z Coordinates on the axis For the first The reflectance at each point.

[0040] A2. In this embodiment of the invention, the effective area of ​​the point cloud data is... That is, the effective area is in the lidar coordinate system. x Shaft length from 0m to 70.4m y Shaft -40m to 40m y Shaft -3m to 1m, such as Figure 2 As shown, point cloud data is obtained within the effective area. :

[0041] In the formula, To effectively divide the network into regions, For the initial point cloud data, This refers to the valid area of ​​the point cloud data.

[0042] A3. Size of the voxel grid in the embodiments of the present invention That is, each voxel grid in x The shaft length is 0.05m. y The shaft length is 0.05m. z The axis length is 0.1m, for point cloud data within the effective area. Divide into ,in, This represents the characteristics of the points contained in each voxel. Represents the coordinates of each voxel. The formula for representing the number of valid points in each voxel is as follows:

[0043] In the formula, For voxel generation networks, The maximum number of sampling points per voxel, in this embodiment of the invention. When the actual number of points exceeds the threshold, the threshold point cloud is retained through random sampling; when the number of points is less than the threshold, zero vector padding is used. The maximum number of voxels per frame, in this embodiment of the invention If the threshold is exceeded, voxels exceeding the limit will be discarded by default.

[0044] A4. At this point, voxel characteristics Including each voxel The characteristics of each point are determined based on the number of valid points in each voxel. By targeting effective points within voxels x axis, y axis, zThe effective point fusion feature within each voxel is obtained by averaging along the axis and reflection intensity dimensions. :

[0045] In the formula, To target effective points within voxels x axis, y axis, z Calculate the mean value along the axis and reflection intensity dimensions. In each voxel Features of each point The number of valid points in each voxel.

[0046] A5. Using a 3D sparse convolutional network as the backbone network for point cloud features to extract sparse tensors of voxel features. The three-dimensional sparse convolutional network consists of multiple sparse convolutional blocks, each of which comprises a sparse convolution and multiple submanifold convolutions. In this embodiment, the subscript 3 represents obtaining the current sparse tensor through three sparse convolutional blocks, as shown in the following formula:

[0047] In the formula, It is a three-dimensional sparse convolutional network. For sparse tensor generation networks, Features fused at effective points within each voxel. The coordinates of each voxel, These are the weights of a 3D sparse convolutional network.

[0048] A6. Acquire initial image data of the surrounding environment. The Swin-Tiny network was used as the image feature extraction network to extract multi-stage image features. A feature pyramid network is used to fuse multi-stage image features. Obtaining fused image features :

[0049]

[0050] In the formula, For the Swin-Tiny network, For initial image data, The weights of the Swin-Tiny network For feature pyramid network, The weights are those of the feature pyramid network.

[0051] B. Integrating point cloud voxel features and image features B1. For sparse tensors For each non-empty voxel, take the coordinates of its three-dimensional center point. As a key point, among them, Indicates the number of non-empty voxels. Indicates the first The coordinates of the three-dimensional center point of a non-empty voxel in the lidar coordinate system The first The central point is x axis, y axis, z Coordinates on the axis. Using pre-calibrated extrinsic and intrinsic parameters of the laser-laser and camera, the coordinates of the voxel center point are... Projecting the image pixel coordinates from the lidar coordinate system to the image pixel coordinate system yields the projected image pixel coordinates. ,in, , For the first The projection points are in the image pixel coordinate system x axis, y The coordinates on the axis are given by the following formula:

[0052] In the formula, Representing the The depth of each projection point K It is the camera intrinsic parameter matrix. It is the extrinsic parameter matrix of the lidar to the camera. , , The first Individual pixel center point in lidar coordinate system x axis, y axis, z Coordinates on the axis.

[0053] B2. In image features Above, for the pixel coordinates of the projected image The corresponding image feature vector is obtained using bilinear interpolation. :

[0054] In the formula, For bilinear interpolation, For image features, These are the pixel coordinates of the projected image.

[0055] B3. Extract the image feature vector With sparse tensors voxel feature vectors The features are then concatenated and then subjected to dimensionality reduction and feature integration through a fully connected layer to obtain the enhanced voxel features. :

[0056] In the formula, It is a fully connected layer. For feature concatenation function, For sparse tensors voxel feature vectors, This is the extracted image feature vector.

[0057] B4. Based on the enhanced voxel characteristics and sparse tensors voxel coordinates New sparse tensors are generated, and three-stage fusion features are extracted using three 3D sparse convolutional networks. , , ,in, The fused features are extracted after a sparse convolutional block. The fused features are extracted from two sparse convolutional blocks. The fusion feature extracted from three sparse convolutional blocks is expressed by the following formula:

[0058]

[0059]

[0060] In the formula, It is a three-dimensional sparse convolutional network. For sparse tensor generation networks, For enhanced voxel features, For sparse tensors voxel coordinates, These are the weights of a 3D sparse convolutional network.

[0061] C. Aggregated multiscale voxel features C1. Features are integrated through a three-stage flattening process in the height direction. ,like Figure 3 As shown, where, 4, 5, and 6 represent the three-stage fusion features extracted by the three 3D sparse convolutional networks to obtain multi-scale BEV features. The formula is as follows:

[0062] In the formula, It is a convolutional neural network. For a network that is flattened in the height direction, The above refers to the three-stage fusion features extracted by three 3D sparse convolutional networks. The weights are the weights of the convolutional neural network, and the BEV features are the bird's-eye view features.

[0063] C2, such as Figure 3 As shown, at each scale, BEV features obtained by upsampling voxels at the current scale are fused with smaller BEV features from the previous scale. This achieves feature aggregation while preserving the spatial structure of the previous scale, reducing information loss of spatial features during the encoding process, fusing features from various scales, and expanding the receptive field. The formula is as follows:

[0064]

[0065] In the formula, The BEV features are the result of multi-scale voxel feature aggregation. , The multi-scale BEV features are obtained by voxel feature compression. It is a convolutional neural network. For upsampling networks, These are the weights of the convolutional neural network.

[0066] C3. BEV features after aggregating multi-scale voxel features Input is fed into the neck and head to decode and output the target's location and category information.

[0067] The basic principles, main features, and advantages of this invention have been described above. Those skilled in the art should understand that this invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of this invention. Various changes and modifications can be made to this invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the invention as claimed.

Claims

1. A point cloud image fusion target detection method based on multi-scale voxel feature aggregation, characterized in that: Includes the following steps: A. Extract point cloud voxel features and image features; B. Integrate point cloud voxel features and image features; C. Aggregated multi-scale voxel features.

2. The point cloud image fusion target detection method based on multi-scale voxel feature aggregation according to claim 1, characterized in that: The method for extracting point cloud voxel features and image features in step A includes the following steps: A1. Collect initial point cloud data of the surrounding environment. ,in, This indicates the number of points in the point cloud of the current frame. Represents the initial features of the point cloud. The first The points are in the lidar coordinate system. x axis, y axis, z Coordinates on the axis For the first Reflectance at each point; A2. Define the valid area for point cloud data as follows: , They are the starting ones x axis, y axis, z The coordinates of the axis, They are terminated respectively x axis, y axis, z The coordinates of the axes are obtained through the effective area of ​​the point cloud data. Acquire point cloud data within the effective area : In the formula, To effectively divide the network into regions, For the initial point cloud data, This is the valid area for point cloud data; A3. Set the size of the voxel mesh. ,in, For voxel mesh in x axis, y axis, z The length on the axis, for point cloud data within the effective area. Divide into ,in, This represents the characteristics of the points contained in each voxel. Represents the coordinates of each voxel. The formula for representing the number of valid points in each voxel is as follows: In the formula, For voxel generation networks, The maximum number of sampling points for each voxel is defined. When the actual number of points exceeds the threshold, a threshold point cloud is retained through random sampling. When the number of points is less than the threshold, zero vector padding is used. This is the maximum number of voxels per frame; if the limit is exceeded, voxels exceeding the limit will be discarded by default. A4. At this point, voxel characteristics Including each voxel The characteristics of each point are determined based on the number of valid points in each voxel. By targeting effective points within voxels x axis, y axis, z The effective point fusion feature within each voxel is obtained by averaging along the axis and reflection intensity dimensions. : In the formula, To target effective points within voxels x axis, y axis, z Calculate the mean value along the axis and reflection intensity dimensions. In each voxel Features of each point The number of valid points in each voxel; A5. Using a 3D sparse convolutional network as the backbone network for point cloud features to extract sparse tensors of voxel features. The formula is as follows: In the formula, It is a three-dimensional sparse convolutional network. For sparse tensor generation networks, Features fused at effective points within each voxel. The coordinates of each voxel, These are the weights of a 3D sparse convolutional network; A6. Acquire initial image data of the surrounding environment. The Swin-Tiny network was used as the image feature extraction network to extract multi-stage image features. A feature pyramid network is used to fuse multi-stage image features. Obtaining fused image features : In the formula, For the Swin-Tiny network, For initial image data, The weights of the Swin-Tiny network For feature pyramid network, The weights are those of the feature pyramid network.

3. The point cloud image fusion target detection method based on multi-scale voxel feature aggregation according to claim 1, characterized in that: Step B describes a method for fusing point cloud voxel features and image features, which includes the following steps: B1. For sparse tensors For each non-empty voxel, take the coordinates of its three-dimensional center point. As a key point, among them, Indicates the number of non-empty voxels. Indicates the first The coordinates of the three-dimensional center point of a non-empty voxel in the lidar coordinate system The first The central point is x axis, y axis, z Coordinates on the axis; using pre-calibrated extrinsic and intrinsic parameter matrices of the laser-radar sensor and camera, the coordinates of the voxel center point are... Projecting the image pixel coordinates from the lidar coordinate system to the image pixel coordinate system yields the projected image pixel coordinates. ,in, , The first The projection points are in the image pixel coordinate system x axis, y The coordinates on the axis are given by the following formula: In the formula, Representing the The depth of each projection point K It is the camera intrinsic parameter matrix. It is the extrinsic parameter matrix of the lidar to the camera. , , The first Individual pixel center point in lidar coordinate system x axis, y axis, z Coordinates on the axis; B2. In image features Above, for the pixel coordinates of the projected image The corresponding image feature vector is obtained using bilinear interpolation. : In the formula, For bilinear interpolation, For image features, These are the pixel coordinates of the projected image; B3. Extract the image feature vector With sparse tensors voxel feature vectors The features are then concatenated and then subjected to dimensionality reduction and feature integration through a fully connected layer to obtain the enhanced voxel features. : In the formula, It is a fully connected layer. For feature concatenation function, For sparse tensors voxel feature vectors, The extracted image feature vector; B4. Based on the enhanced voxel characteristics and sparse tensors voxel coordinates New sparse tensors are generated, and three-stage fusion features are extracted using three 3D sparse convolutional networks. , , ,in, The fused features are extracted through a 3D sparse convolutional network. The fused features are extracted from two 3D sparse convolutional networks. The fused features extracted by three 3D sparse convolutional networks are expressed in the following formula: In the formula, It is a three-dimensional sparse convolutional network. For sparse tensor generation networks, For enhanced voxel features, For sparse tensors voxel coordinates, These are the weights of a 3D sparse convolutional network.

4. The point cloud image fusion target detection method based on multi-scale voxel feature aggregation according to claim 1, characterized in that: The method for aggregating multi-scale voxel features described in step C includes the following steps: C1. Features are integrated through a three-stage flattening process in the height direction. ,in, , , This indicates that the three-stage fusion features extracted by the above three 3D sparse convolutional networks are used to obtain multi-scale BEV features. The formula is as follows: In the formula, It is a convolutional neural network. For a network that is flattened in the height direction, The above refers to the three-stage fusion features extracted by three 3D sparse convolutional networks. The weights are those of the convolutional neural network, and the BEV features are bird's-eye view features. C2. At each scale, BEV features obtained by upsampling voxels at the current scale are fused with smaller BEV features from the previous scale, as shown in the following formula: In the formula, The BEV features are the result of multi-scale voxel feature aggregation. , The multi-scale BEV features are obtained by voxel feature compression. It is a convolutional neural network. For upsampling networks, These are the weights of the convolutional neural network; C3. BEV features after aggregating multi-scale voxel features Input is fed into the neck and head to decode and output the target's location and category information.

Images