A target key point-based point cloud and image feature fusion method

Abstract

The application discloses a point cloud and image feature fusion method based on target key points, a target key point decoder is used to jointly decode a group of target key points from image and point cloud data, voxelization processing and voxel feature coding are performed on the key points, the obtained key point voxel features and voxel features coded from the point cloud are sent into a multi-modal voxel feature fusion module, and through an adaptive fusion mode, reinforced multi-modal voxel features are obtained, and the generated multi-modal voxel features can be combined with various detection heads to meet different requirements in actual application.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a method for fusing point cloud and image features based on target key points. Background Technology

[0002] With the continuous development of sensor technology and the increasing task demands in various application fields, utilizing multiple sensors to provide accurate and reliable multimodal data input for practical applications has become a widely adopted technical approach. Especially for point cloud and image data, the complementary fusion of the two plays a crucial role in fields such as medicine, transportation, and robotics. Point clouds describe the actual distribution of objects in space as discrete points, while images record the forward projection of objects in space through a two-dimensional array. Besides having different data formats, due to sensor resolution limitations, the pixel density of images typically far exceeds the number of points contained in point clouds. Therefore, how to leverage the respective advantages of point cloud and image data, and efficiently and accurately fuse point cloud and image features with different representations and data densities, presents a significant challenge to existing multimodal data fusion methods. Furthermore, current fusion methods generally employ a feature fusion approach that passively fuses image information with point cloud as the primary driver. However, this approach wastes the rich target information in the image, limits the fusion capability of images and point clouds, and thus impacts downstream tasks. Therefore, developing a high-quality point cloud and image feature fusion method is extremely urgent. Summary of the Invention

[0003] To overcome the shortcomings of existing technologies, this invention provides a point cloud and image feature fusion method based on target key points. A target key point decoder jointly decodes a set of target key points from image and point cloud data. Then, these key points are voxelized and voxel feature encoded. The obtained key point voxel features and the voxel features encoded from the point cloud are then fed into a multimodal voxel feature fusion module. In this module, an adaptive fusion method is used to obtain enhanced multimodal voxel features. The generated multimodal voxel features can be combined with various detection heads to meet different needs in practical applications.

[0004] The technical solution adopted by this invention to solve its technical problem is as follows:

[0005] Step 1: Construct a point cloud and image feature fusion system, including a target keypoint decoder and a multimodal voxel feature fusion module;

[0006] The target key point decoder is used to jointly decode target key points from image features and point cloud features and extract corresponding key point features, so as to represent image information with key points and then participate in the subsequent deep fusion process with point cloud features.

[0007] The multimodal voxel feature fusion module is used to voxelize the generated key points and encode voxel features, and adaptively fuse the obtained key point voxel features with the point cloud voxel features to obtain fused multimodal voxel features.

[0008] Step 2: The target key point decoder uniformly initializes a set of target key points in the point cloud space, and then uses image and point cloud information to gradually update the position and features of the key points through multiple decoding layers. After multiple iterations of the position and features of the key points through these decoding layers, the target key points representing image information are finally generated.

[0009] Step 3: The multimodal voxel feature fusion module voxelizes the obtained key points, then encodes the key point features using a voxel feature encoder, and adaptively fuses the obtained key point voxel features with the voxel features from the point cloud to obtain enhanced multimodal voxel features.

[0010] Furthermore, the target keypoint decoder is used to jointly decode a set of keypoints from image features and point cloud features;

[0011] Step 2-1: Use ResNet-50 and Feature Pyramid Network to extract image features, and then use the obtained image features at five scales, which are 1 / 4, 1 / 8, 1 / 16, 1 / 32 and 1 / 64 of the original image size, to provide image information for the target part;

[0012] Step 2-2: The target key point decoder is located within the point cloud range [(x L ,x H ),(y L ,y H ),(z L ,z H A set of target key points N are uniformly initialized. x ×N y ×N z The initial keypoints are placed with step sizes (x, y, and z) on the x, y, and z axes, respectively. H -x L ) / N x , (y H -y L ) / N y and (Z) H -Z L ) / N zThese keypoints are then projected onto the image, and their corresponding 2D features at the projection locations are used as the initial features of the keypoints. Next, in any decoding layer of the target keypoint decoder, PointNet building blocks are used to learn the point cloud features of the vicinity for each target keypoint. Then, based on deformable DETR, the product of the initial features and the point cloud features of the keypoints is used as the query value for the target keypoints. These query values ​​are then used to retrieve image information related to the target. The keypoint features enhanced by deformable attention not only serve as the initial features q of the keypoints in the next decoding layer, but also... l Furthermore, an additional regression layer is used to generate keypoint offsets to update the keypoint position information. These newly generated keypoints will iteratively use the above method to update the keypoint position and feature information in the next decoding layer; the process is shown in formula (1):

[0013]

[0014] in, Let F represent the deformable DETR of the (l+1)th layer, P represent the set of the original point clouds, and F represent the deformable DETR of the (l+1)th layer. img Represents multi-scale image features;

[0015] Steps 2-3: As shown in formula (2), the updated keypoint positions in each decoding layer are encoded through a linear layer to provide keypoint position information. This position information is then added to the output of the current keypoint decoding layer, and the resulting output is used as the final output of the current decoding layer.

[0016]

[0017] in, It contains a linear layer, a normalized layer, and a ReLU activation function; reg(.) contains a linear layer and a ReLU activation function.

[0018] Steps 2-4: After multiple decoding layer iterations, the target keypoint decoder outputs the final position and features of the target keypoints. The decoded keypoint features are then passed through two linear layers and an activation function ReLU to generate the foreground scores of the target keypoints. The generated foreground scores are used to calculate the loss between the foreground scores and the corresponding labels using the cross-entropy loss function. The foreground labels of the target keypoints are obtained by determining the relative position of the keypoint to the nearest target proposal box. For keypoints located inside the target candidate proposal box, they are assigned a foreground label with a score of 1; otherwise, they are assigned a foreground label with a target component score of 0.

[0019] Furthermore, the multimodal voxel feature fusion module is used to voxelize the generated key points and encode voxel features, and then adaptively fuse the obtained key point voxel features with the voxel features of the point cloud to generate enhanced multimodal voxel features.

[0020] Step 3-1: After obtaining the key points, voxelization and voxel feature encoding are performed on all key points. The encoded key point voxel features are adaptively fused with the point cloud voxel features. The weights of each participating in the fusion are obtained by adding the two features and then passing them through a linear layer and a tanh activation function respectively.

[0021] Step 3-2: For keypoint voxels and point cloud voxels with the same spatial location, use an adaptive fusion method for fusion; for point cloud voxels that lack matching keypoint voxels, search for already matched point cloud voxels within a spherical neighborhood with a radius of 0.5m. If a matched point cloud voxel is found, its corresponding keypoint voxel is used as the keypoint voxel that should be matched by the currently searched point cloud voxel, and then its corresponding keypoint voxel is fused with its own point cloud voxel using the adaptive fusion method; for point cloud voxels that have not been matched with keypoint voxels in the neighborhood, the original point cloud voxel features are directly used as the final voxel features.

[0022] The beneficial effects of this invention are as follows:

[0023] 1. This invention jointly decodes target key points from image features and point cloud features to fully utilize the semantic features in the image and the geometric features in the point cloud, thereby generating high-quality target key points, which helps to improve the recall rate of target detection.

[0024] 2. This invention uses target key points to represent image information, which greatly enhances the feature fusion process of image-dominated point clouds and images. At the same time, the obtained multimodal features can effectively improve the detection performance of downstream networks. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of a point cloud and image feature fusion method based on target key points.

[0026] Figure 2 This is a schematic diagram of the decoding layer in the target keypoint decoder. Detailed Implementation

[0027] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0028] To address the feature fusion problem between images and point clouds, this invention proposes a point cloud and image feature fusion method based on target key points. It uses a target key point decoder to jointly decode a set of target key points from image and point cloud data. These key points are then voxelized and their features encoded. The obtained key point voxel features, along with the voxel features encoded from the point cloud, are fed into a multimodal voxel feature fusion module. An adaptive fusion method is used to obtain enhanced multimodal voxel features. The generated multimodal voxel features can be combined with various detection heads to meet different needs in practical applications.

[0029] The present invention adopts the following technical solution:

[0030] The target keypoint decoder is used to jointly decode target keypoints from image features and point cloud features and extract corresponding keypoint features, so as to represent image information with keypoints and then participate in the subsequent deep fusion process with point cloud features.

[0031] The multimodal voxel feature fusion module is used to voxelize the generated key points and encode voxel features. The obtained key point voxel features are adaptively fused with point cloud voxel features to obtain fused multimodal voxel features. The fused voxel features can be used for downstream tasks.

[0032] The target keypoint decoder initializes a set of target keypoints uniformly in the point cloud space. Then, through a keypoint decoder, the position and features of the keypoints are gradually updated using image and point cloud information in multiple decoding layers. After multiple iterations of the position and features of the keypoints through these decoding layers, the target keypoints representing image information are finally generated. These keypoints can be fused with the original point cloud data for deeper feature fusion.

[0033] The multimodal voxel feature fusion module voxels the obtained key points and then encodes the key point features using a voxel feature encoder. The obtained key point voxel features are adaptively fused with voxel features from the point cloud to obtain enhanced multimodal voxel features. These fused multimodal voxel features are generated by guiding the fusion of multimodal voxel features with image information as the main factor.

[0034] Example:

[0035] This invention provides a method for fusing point cloud and image features based on target key points, such as... Figure 1 As shown, it includes:

[0036] A target keypoint decoder is used to jointly decode a set of keypoints from image features and point cloud features. In implementation, a ResNet-50 and a feature pyramid network are used to extract image features, and then the obtained image features at five scales (1 / 4, 1 / 8, 1 / 16, 1 / 32, and 1 / 64 of the original image size) are used to provide image information for the target part. Figure 2 As shown, the target keypoint decoder is located within the point cloud range [(x L ,x H ),(y L ,y H ),(z L ,z H A set of target key points N are uniformly initialized. x ×N y ×N z The initial keypoints are placed with step sizes (x, y, and z) on the x, y, and z axes, respectively. H -x L ) / N x , (y H -y L ) / N y and (Z) H -Z L ) / N z Then, these key points are projected onto the image, and their corresponding two-dimensional features at the projection positions are used as the initial features of the key points. Then, in any decoding layer of the target key point decoder, the PointNet building blocks are used to learn the point cloud features of the vicinity of each target key point. Then, based on the deformable DETR, the product of the initial features of the key points and the point cloud features of the key points is used as the query value of the target key points. These query values ​​are then used to query image information related to the target. The key point features enhanced by deformable attention not only serve as the initial features ql of the key points in the next decoding layer, but also generate key point offsets through a regression layer to update the position information of the key points. These newly generated key points will iteratively use the above method to update the position and feature information of the key points in the next decoding layer. The above decoding process is shown in Equation (1):

[0037]

[0038] Let F represent the deformable DETR of the (l+1)th layer, P represent the set of the original point clouds, and F represent the deformable DETR of the (l+1)th layer. img This represents multi-scale image features. Furthermore, as shown in Equation (2), the updated keypoint locations in each decoding layer are encoded through a linear layer to provide keypoint location information, and this location information is added to the output of the current keypoint decoding layer. The resulting output is then used as the final output of the current decoding layer.

[0039]

[0040] After multiple decoding layer iterations, the target keypoint decoder outputs the final position and features of the target keypoints. The decoded keypoint features are then passed through two linear layers and an activation function ReLU to generate foreground scores for the target keypoints. The generated foreground scores are used to calculate the loss between the foreground scores and their corresponding labels using the cross-entropy loss function. The foreground labels for the target keypoints are obtained by determining the relative position of the keypoint to the nearest target proposal box. Keypoints located inside the target proposal box are assigned a foreground label with a score of 1; otherwise, they are assigned a foreground label with a target part score of 0.

[0041] The multimodal voxel feature fusion module is used to voxelize the generated key points and encode voxel features, and then adaptively fuse the obtained key point voxel features with the voxel features of the point cloud to generate enhanced multimodal voxel features.

[0042] In implementation, after obtaining the key points, all key points are voxelized and voxel features are encoded. The encoded key point voxel features are adaptively fused with the point cloud voxel features. The weights of each participating in the fusion are obtained by adding the two features and then passing them through a linear layer and a tanh activation function respectively. Since the number of keypoint voxels is far less than the number of point cloud voxels, only a small number of point cloud voxels can have matching keypoint voxels during the fusion process. For keypoint voxels and point cloud voxels with the same spatial location, the adaptive fusion method is directly used for fusion. For point cloud voxels that lack matching keypoint voxels, a search is conducted within a spherical neighborhood with a radius of 0.5m for already matched point cloud voxels. If a matched point cloud voxel is found, its corresponding keypoint voxel is used as the keypoint voxel that should be matched with the currently searched point cloud voxel. Then, its corresponding keypoint voxel is fused with its own point cloud voxel using the adaptive fusion method. For point cloud voxels that have not been found to have matched keypoint voxels in the neighborhood, the original point cloud voxel features are directly used as the final voxel features.

Claims

1. A method for fusing point cloud and image features based on target key points, characterized in that, Includes the following steps: Step 1: Construct a point cloud and image feature fusion system, including a target keypoint decoder and a multimodal voxel feature fusion module; The target key point decoder is used to jointly decode target key points from image features and point cloud features and extract corresponding key point features, so as to represent image information with key points and then participate in the subsequent deep fusion process with point cloud features. The multimodal voxel feature fusion module is used to voxelize the generated key points and encode voxel features, and adaptively fuse the obtained key point voxel features with the point cloud voxel features to obtain fused multimodal voxel features. Step 2: The target key point decoder uniformly initializes a set of target key points in the point cloud space, and then uses image and point cloud information to gradually update the position and features of the key points through multiple decoding layers. After multiple iterations of the position and features of the key points through these decoding layers, the target key points representing image information are finally generated. The target keypoint decoder is used to jointly decode a set of keypoints from image features and point cloud features; Step 2-1: Use ResNet-50 and Feature Pyramid Network to extract image features, and then use the obtained image features at five scales, which are 1 / 4, 1 / 8, 1 / 16, 1 / 32 and 1 / 64 of the original image size, to provide image information for the target part; Step 2-2: The target key point decoder is located within the point cloud range [( ),( ),( Uniformly initialize a set of target key points The initial keypoints are placed with step sizes on the x, y, and z axes respectively. , and These keypoints are then projected onto the image, and their corresponding 2D features at the projection locations are used as the initial features of the keypoints. Next, in any decoding layer of the target keypoint decoder, PointNet building blocks are used to learn the point cloud features of the vicinity for each target keypoint. Then, based on deformable DETR, the product of the initial features and the point cloud features of the keypoints is used as the query value for the target keypoints. These query values ​​are then used to retrieve image information related to the target. The keypoint features enhanced by deformable attention not only serve as the initial features of the keypoints in the next decoding layer but also... Furthermore, an additional regression layer is used to generate keypoint offsets to update the keypoint position information. These newly generated keypoints will be iteratively updated with the above method in the next decoding layer to update the keypoint position and feature information. The process is shown in formula (1): （1） in, Let represent the deformable DETR of the (l+1)th layer, and P represent the set of the original point clouds. Represents multi-scale image features; Steps 2-3: As shown in formula (2), the updated keypoint positions in each decoding layer are encoded through a linear layer to provide keypoint position information. This position information is then added to the output of the current keypoint decoding layer, and the resulting output is used as the final output of the current decoding layer. （2） in, It contains a linear layer, a normalized layer, and a ReLU activation function. It contains a linear layer and a ReLU activation function; Steps 2-4: After multiple decoding layer iterations, the target keypoint decoder outputs the final position and features of the target keypoints; the decoded keypoint features are then passed through two linear layers and an activation function ReLU to generate the foreground scores of the target keypoints. The generated foreground scores are used to calculate the loss between the foreground scores and the corresponding labels using the cross-entropy loss function. The foreground labels of the target keypoints are obtained by determining the relative position of the keypoint to the nearest target proposal box. For keypoints located inside the target candidate proposal box, they are assigned a foreground label with a score of 1; otherwise, they are assigned a foreground label with a target component score of 0. Step 3: The multimodal voxel feature fusion module voxelizes the obtained key points, then encodes the key point features using a voxel feature encoder, and adaptively fuses the obtained key point voxel features with the voxel features from the point cloud to obtain enhanced multimodal voxel features.

2. The point cloud and image feature fusion method based on target key points according to claim 1, characterized in that, The multimodal voxel feature fusion module is used to voxelize the generated key points and encode voxel features, and then adaptively fuse the obtained key point voxel features with the voxel features of the point cloud to generate enhanced multimodal voxel features. Step 3-1: After obtaining the key points, voxelization and voxel feature encoding are performed on all key points. The encoded key point voxel features are adaptively fused with the point cloud voxel features. The weights of each participating in the fusion are obtained by adding the two features and then passing them through a linear layer and a tanh activation function respectively. Step 3-2: For keypoint voxels and point cloud voxels with the same spatial location, an adaptive fusion method is used for fusion. For point cloud voxels that lack matching keypoint voxels, a search is conducted within a spherical neighborhood with a radius of 0.5m to find matching point cloud voxels. If a matching point cloud voxel is found, its corresponding keypoint voxel is used as the keypoint voxel that should be matched by the currently searched point cloud voxel. Then, its corresponding keypoint voxel is fused with its own point cloud voxel using the adaptive fusion method. For point cloud voxels that do not find a matching keypoint voxel in the neighborhood, the original point cloud voxel features are directly used as the final voxel features.

Images