A 3D Single-Target Tracking Method Based on Multimodal Information Fusion

Abstract

This invention provides a 3D single-target tracking method based on multimodal information fusion. First, it achieves spatial alignment between the original image and laser point cloud. Second, it constructs a dual-stream feature extraction network based on deep learning to extract high-dimensional semantic features from the image and point cloud in parallel. Then, it weights important information within each modality using a self-attention mechanism and constructs semantic connections between different modalities using a cross-attention mechanism. Next, it calculates texture and geometric similarity based on semantically enhanced features and generates multimodal similarity features using an attention mechanism. Finally, it uses a multi-layer convolutional structure to predict the target's spatial position and orientation. This invention aims to fully utilize the complementary advantages of heterogeneous data from multiple sensors and adaptively fuse image texture information and point cloud geometric features using the attention mechanism in deep learning to generate more robust multimodal features. This allows for accurate regression of the target's 3D coordinates and orientation, improving the robustness and accuracy of the 3D target tracker.

Description

Technical Field

[0001] This invention belongs to the field of three-dimensional point cloud single-target tracking technology, specifically relating to a three-dimensional single-target tracking method based on multimodal information fusion. Background Technology

[0002] In recent years, with the rapid development of artificial intelligence technology, intelligent equipment such as robots and autonomous vehicles have gradually appeared in the public eye and become a new strategic direction for "Made in China." The safe and stable operation of these intelligent products is inseparable from environmental perception technology. Environmental perception mainly refers to using sensors such as cameras, lidar, and ultrasonic sensors to acquire comprehensive environmental status data, and then using computer processors and intelligent algorithms to extract the most critical semantic information, such as the location and size of nearby vehicles and pedestrians. This information provides crucial data support for the decision-making and control of intelligent systems, ensuring the reliable operation of the systems.

[0003] In recent years, the rapid rise of deep learning has greatly promoted the development of 3D target tracking algorithms. Many researchers have tried to apply deep learning methods to target tracking and have achieved remarkable results. Compared with traditional algorithms, the high-dimensional features extracted by deep learning are more suitable for matching targets, thus achieving robust target tracking. However, most existing 3D target tracking algorithms inherit the Siamese network structure of 2D tracking algorithms and rely on the geometric information provided by point clouds to construct similarity. This makes it difficult for trackers to distinguish objects with similar structures. The texture information of images can effectively solve this problem. For example, texture features such as appearance and clothing can distinguish two people with similar geometric structures. Therefore, it is possible to consider using deep learning methods to achieve deep fusion of multimodal data to solve the problem of trackers easily failing in complex environments.

[0004] Chinese patent CN114862911A, "A Single-Target Tracking Method for 3D Point Clouds Based on Graph Convolution," proposes a single-target tracking method for 3D point clouds based on graph convolution. This invention improves upon existing P2B (Point-to-Browse) 3D single-target tracking methods. First, a template and a search point cloud are used as network inputs. Second, the point cloud is downsampled and seed point features are extracted. Then, a graph convolution module is used to fuse global and local features, and template cues are encoded into the search region. Finally, the seed points encoding the template information are input into a Hough voting module to locate and track the target within the search region and generate its 3D bounding box. This technical solution uses lidar as a sensor to perceive the environment. Although the point cloud data collected by the lidar can describe the geometric contours of objects, the sparsity and disorder of point clouds mean that the object represented by the point cloud is incomplete at long distances or under occlusion, which can lead to a sharp decline in tracker performance. Furthermore, since this solution relies solely on the geometric information of the point cloud, the tracker will have difficulty distinguishing the target when the geometric structures of the target and the background are similar. This means that the method is prone to target confusion and loss, making it difficult to guarantee good tracking performance in complex scenarios.

[0005] Chinese patent CN111091582A, "A Single-Vision Target Tracking Algorithm and System Based on Deep Neural Networks," proposes a single-vision target tracking algorithm and system based on deep neural networks. This invention first specifies the target to be tracked in the initial frame of an image sequence, providing prior information about its position and size. Then, in subsequent frames, a convolutional network with shared weights extracts the target template and search region features. Finally, a convolutional and foreground-background discrimination network regresses the target bounding box height, width, and center position. Furthermore, the invention proposes a size adjustment module that dynamically adjusts the cropping size of the search region according to the target size, thus adapting to targets of different sizes and motion characteristics. This method uses images as the sole data source. While images possess rich texture information, they are easily affected by environmental factors such as lighting and weather, thus limiting the image tracker's all-weather operation capability. Moreover, because images lack three-dimensional information about objects, image trackers can only track targets in a two-dimensional plane. However, in tracking tasks in robotics and autonomous driving, capturing the target's motion trajectory in three-dimensional space is more desirable than estimating its positional changes in the image plane, thus limiting the application scenarios of image trackers. Summary of the Invention

[0006] To address the aforementioned issues, this invention presents a 3D single-target tracking method based on multimodal information fusion. It aims to fully leverage the complementary advantages of heterogeneous data from multiple sensors and adaptively fuse image texture information and point cloud geometric features using the attention mechanism in deep learning. This generates more robust multimodal features to accurately regress the 3D coordinates and orientation of the target, solving the problem of point cloud trackers struggling to distinguish structurally similar objects. This improves the network's ability to track distant targets and mitigates the impact of point cloud sparsity and occlusion on tracking performance, ultimately enhancing the robustness and accuracy of the 3D target tracker. Furthermore, by using images and low-line-count LiDAR data as system inputs and providing robust tracking results, it avoids reliance on expensive high-line-count LiDAR for tracking, thus reducing vehicle production costs and facilitating the rapid deployment of autonomous driving products, resulting in substantial economic benefits.

[0007] A three-dimensional single-target tracking method based on multimodal information fusion includes:

[0008] Step 1: Spatial alignment of images and point cloud data acquired from different sensors; including:

[0009] Step 1.1: Project the point cloud onto the image plane based on the camera's intrinsic and extrinsic parameters;

[0010] Step 1.2: Assign the point cloud depth to the corresponding pixels;

[0011] Step 1.3: Back-project the pixels into three-dimensional space to generate a pseudo point cloud that has image texture information and is aligned with the original point cloud coordinate system;

[0012] Step 2: Construct a two-stream feature extraction network based on deep learning methods to achieve parallel extraction of high-dimensional semantic features from pseudo-point clouds and point clouds by different network branches; including:

[0013] Step 2.1: Extract texture features from the pseudo-point cloud; including:

[0014] Step 2.1.1: For the pseudo-point cloud obtained in Step 1, process the template pseudo-point cloud. and search for pseudo-point clouds The farthest point sampling algorithm was used to downsample the point cloud, and Q points were obtained as key points respectively.

[0015] Step 2.1.2: For the template pseudo-point cloud and search pseudo-point cloud obtained in Step 2.1.1, take Q key points as the center, perform KNN (K nearest neighbor) clustering on the points within the radius R, and then use MLP (Multilayer Perceptron) network to aggregate the clustered point features into the key points;

[0016] Step 2.1.3: Repeat the application of farthest point sampling, KNN clustering, and MLP network to reduce the number of keypoints to N, and output a template point set containing N points. and search point set Each s i Given a three-dimensional coordinate vector c i and a C-dimensional descriptor f that represents the local texture information of an object i Composition, i.e., s i =(c i ,f i );

[0017] Step 2.2: Extract the geometric features of the real point cloud, including:

[0018] Step 2.2.1: For the real point cloud, convert the template point cloud... and search point cloud Voxelization is performed to convert it into a dense voxel representation;

[0019] Step 2.2.2: For the template point cloud and search point cloud Three-dimensional sparse convolutional networks are used to extract geometric features of point clouds from voxels to generate template point clouds. Three-dimensional voxel features Search point cloud Three-dimensional voxel features In the formula, W, L, H, and C represent the dimensions of the corresponding tensors;

[0020] Step 2.2.3: Targeting 3D voxel features The height channel and feature channel are merged separately to output denser BEV (bird's-eye view) features. Where c′=H×C;

[0021] Step 3: Combine self-attention and cross-attention to achieve multimodal feature interaction and enhancement; including:

[0022] Step 3.1: Construct long-distance dependencies within a modality through a self-attention mechanism, learn the importance of different features, and weight the features accordingly;

[0023] Step 3.2: Construct semantic connections between different modalities through cross-attention, so that information from different modalities can strengthen the features of the same semantic object, thereby generating more robust multimodal semantic features;

[0024] Step 4: Cross-modal modeling capability based on attention mechanism, fusing the similarity between image pixels and point cloud data; including:

[0025] Step 4.1: Apply a pixel-wise cosine similarity calculation function to the semantic enhancement features of the search point cloud and the template point cloud to generate a geometric similarity S. geo ∈R W×L×D In the formula, W×L×D represents the tensor S geo The dimension;

[0026] Step 4.2: Apply a point-by-point cosine similarity calculation function to the semantic enhancement features of the search pseudo-point cloud and the template pseudo-point cloud to generate texture similarity S. tex ∈R N×D In the formula, N×D represents the tensor S tex The dimension;

[0027] Step 4.3: Using a cross-attention mechanism, geometric similarity and texture similarity are fused to generate a more robust multimodal similarity feature S′. fus ;

[0028] Step 5: Based on the multimodal similarity feature S′ fus It employs a multi-layer convolutional structure to predict the spatial location and orientation of a target; including:

[0029] Step 5.1: Extract target cues hidden in multimodal similarity features through a multi-layer convolutional structure to generate feature maps;

[0030] Step 5.2: Predict the target's confidence level and corresponding spatial attributes at each location in the feature map. The location with the highest confidence level is the target's undetermined location. The spatial attributes are the target's position deviation and orientation, which are used to correct the undetermined location.

[0031] Step 5.3: Use the corrected undetermined position as the prediction result for the tracking target.

[0032] The beneficial effects of this invention are:

[0033] 1) This invention can make full use of the characteristics of each sensor to achieve complementary advantages, improve the tracker's ability to track distant targets or occlusion situations, and can better distinguish objects with similar structures.

[0034] 2) Compared with existing multimodal trackers, the tracking method proposed in this invention achieves the best performance in terms of both success rate and accuracy;

[0035] 3) This method can be deployed on mobile robot platforms or autonomous vehicles and has real-time operation capability. Specifically, taking the Nvidia 2080Ti GPU as an example, this method can achieve a processing speed of 12FPS. Attached Figure Description

[0036] Figure 1This is a schematic diagram of the three-dimensional single-target tracking method based on multimodal information fusion in this invention.

[0037] Figure 2 This is a diagram of the multimodal dual-stream feature extraction network structure in this invention;

[0038] Figure 3 This is a diagram of the multimodal feature interaction and enhancement network structure in this invention;

[0039] Figure 4 This is a diagram of the multimodal similarity fusion structure in this invention. Detailed Implementation

[0040] The invention will be further explained below with reference to the accompanying drawings and specific implementation examples.

[0041] This invention focuses on a crucial task in environmental perception—3D target tracking. The core of this task is to estimate the target's subsequent motion state based on its initial state and data from the preceding and following frames (images, point clouds). Since perception systems often require long-term positioning of specific targets in practical applications, 3D target tracking has extremely wide application value. Its application areas include: (1) Autonomous driving, using lidar and cameras to automatically monitor and track pedestrians, vehicles, and other targets in the environment surrounding the vehicle, ensuring a safe distance from vehicles, pedestrians, and various objects, and effectively avoiding potential collision risks caused by emergencies; (2) Intelligent monitoring, intelligently analyzing monitoring scenarios and continuously tracking suspicious targets, providing early warnings of abnormal activities by suspects, thereby effectively preventing dangerous situations from occurring; (3) Military guidance, using computing devices to intelligently identify targets of strategic value, achieving precise positioning, continuous tracking, and effective strikes.

[0042] The design principle diagram of this invention is as follows: Figure 1As shown, the invention first performs spatial alignment on the multimodal data. This involves projecting pixels containing texture information into 3D space using camera intrinsic and extrinsic parameters, generating a pseudo-point cloud that exists in the same physical space as the real point cloud. Prior information (3D position and size) is then obtained from the target ground truth box in the initial frame. The target template region and potential target search regions are then extracted from the point cloud and pseudo-point cloud. Next, a two-stream network is used to extract features from two different modalities. The pseudo-point cloud branch is used to quickly encode texture information, while the point cloud branch extracts geometric features and generates a dense BEV representation. Subsequently, an attention interaction and enhancement module weights important features within each modality and constructs semantic associations between important features from different modalities, resulting in more robust multimodal features. Furthermore, the enhanced features are used to calculate intramodal similarity to generate texture similarity and geometric similarity, which are then adaptively fused using an attention mechanism. Finally, the invention uses multimodal similarity features to predict the target's position and orientation.

[0043] A 3D single-target tracking method based on multimodal information fusion is proposed. First, spatial alignment of the original image and laser point cloud is achieved. Second, a dual-stream feature extraction network is constructed based on deep learning to extract high-dimensional semantic features from the image and point cloud in parallel. Then, important information within each modality is weighted using a self-attention mechanism, and semantic connections between different modalities are constructed using a cross-attention mechanism. Next, texture and geometric similarity are calculated based on semantically enhanced features, and multimodal similarity features are generated using an attention mechanism. Finally, a multi-layer convolutional structure is used to predict the target's spatial location and orientation. The specific implementation process is as follows:

[0044] Image and point cloud data spatial alignment: Since images and point clouds are located in different sensor coordinate systems, directly using neural networks to construct the correspondence between different data is simple and crude, which can easily lead to confusion in the fusion of different semantic information. Therefore, this invention adopts the method of mapping image pixels to point cloud space to ensure the spatial alignment of multimodal data.

[0045] Step 1: Spatial alignment of images and point cloud data acquired from different sensors; including:

[0046] Step 1.1: Project the point cloud onto the image plane based on the camera's intrinsic and extrinsic parameters;

[0047] Step 1.2: Assign the point cloud depth to the corresponding pixels;

[0048] Step 1.3: Back-project the pixels into three-dimensional space to generate a pseudo point cloud that has image texture information and is aligned with the original point cloud coordinate system;

[0049] Assuming the three-dimensional coordinates of a spatial point P obtained by the lidar are X = (x, y, z), the rotation matrix between the lidar and the camera coordinate system is R, the translation matrix is ​​T, and the camera intrinsic parameter matrix is ​​K, then the pixel Y = (u, v) that uniquely corresponds to the spatial point P in the pixel plane can be obtained according to the following point cloud-image projection formula, where (u, v) are pixel coordinates.

[0050]

[0051] Where (f x ,f y (c) is the focal length. x ,c y The origin of the image is (u,v). Next, the point cloud depth Z is assigned to the corresponding image pixel (u,v). Then, the pixel is back-projected into the point cloud coordinate system according to the point cloud-image projection formula mentioned above to obtain a pseudo point cloud with image texture information.

[0052] Image and point cloud dual-stream feature extraction: Obtaining a pseudo-point cloud from the image that is in the same spatial coordinate system as the point cloud, i.e. (Template pseudo-point cloud) and (Search for pseudo-point clouds), then transfer the real point cloud. (Template dot cloud) (Search point cloud) and pseudo point cloud are simultaneously input into a dual-stream feature extraction network to extract features of different modalities.

[0053] Step 2: Construct a two-stream feature extraction network based on deep learning methods to achieve parallel extraction of high-dimensional semantic features from pseudo-point clouds and point clouds by different network branches; including:

[0054] Step 2.1: Extract texture features from the pseudo-point cloud; including:

[0055] Step 2.1.1: For the pseudo-point cloud obtained in Step 1, process the template pseudo-point cloud. and search for pseudo-point clouds The farthest point sampling algorithm was used to downsample the point cloud, and Q points were obtained as key points respectively.

[0056] Step 2.1.2: For the template pseudo-point cloud and search pseudo-point cloud obtained in Step 2.1.1, take Q key points as the center, perform KNN (K nearest neighbor) clustering on the points within the radius R, and then use MLP (Multilayer Perceptron) network to aggregate the clustered point features into the key points;

[0057] Step 2.1.3: Repeat the application of farthest point sampling, KNN clustering, and MLP network to reduce the number of keypoints to N, and output a template point set containing N points. and search point set Each s iGiven a three-dimensional coordinate vector c i and a C-dimensional descriptor f that represents the local texture information of an object i Composition, i.e., s i =(c i ,f i );

[0058] Step 2.2: Extract the geometric features of the real point cloud, including:

[0059] Step 2.2.1: For the real point cloud, convert the template point cloud... and search point cloud Voxelization is performed to convert it into a dense voxel representation;

[0060] Step 2.2.2: For the template point cloud and search point cloud Three-dimensional sparse convolutional networks are used to extract geometric features of point clouds from voxels to generate template point clouds. Three-dimensional voxel features Search point cloud Three-dimensional voxel features In the formula, W×L×H×C represents the tensor dimension.

[0061] Step 2.2.3: Targeting 3D voxel features The height channel and feature channel are merged separately to output a denser BEV (bird's-eye view) feature. Where C′=H×C;

[0062] like Figure 2 As shown, in the pseudo-point cloud branch, the pseudo-point cloud template is first... and search point cloud The farthest point sampling algorithm is applied, and the sampling results are then clustered to obtain Q cluster centers as key points in the point cloud. Points within a radius R are then clustered using KNN (K-nearest neighbor) clustering, and finally, an MLP (Multilayer Perceptron) network is used to aggregate the features of the clustered points into the key points. This process is repeated multiple times.

[0063]

[0064]

[0065] The pseudo-point cloud branch ultimately outputs a template point set containing N points. and search point set Each s i Given a three-dimensional coordinate vector c i and a C-dimensional descriptor f that represents the local texture information of an object i Composition s i =(ci ,f i To simplify the formula, the following will use... and replace and Features of the pseudo-point cloud template region, This branch of the feature extraction method can quickly aggregate the texture features of the pseudo-point cloud, and finally rely only on a small number of points to represent the local texture features of the background and foreground, effectively reducing the computational cost of subsequent networks.

[0066] Unlike pseudo-point cloud branches, real point cloud branches first apply a template. and search point cloud Voxelization is performed to transform sparse point clouds into dense voxels. Then, 3D sparse convolution is used to extract the geometric features of the point cloud in the voxels and generate template 3D voxel features. and searching for three-dimensional voxel features Then The height channel and feature channel are merged to output a denser template BEV feature. and searching BEV features Where C′=H×C, this method helps to mitigate the impact of point cloud sparsity.

[0067] Multimodal Feature Interaction and Enhancement: Obtaining Pseudo-Point Cloud Template Features Pseudo-point cloud search features and template features of real point clouds Real point cloud search features Each has its own attention enhancement features.

[0068] First, a self-attention mechanism is used to assign weights to the point cloud and pseudo-point cloud features, without changing the feature sizes during this process. Let F be the input feature, and after processing by the self-attention mechanism, the output is an attention-enhanced feature. Then Furthermore, a cross-attention mechanism is used to construct semantic associations across different modalities, outputting multimodal semantic enhancement features. The following will first introduce the basic principles of the attention mechanism:

[0069] The attention mechanism consists of three parts: input feature encoding, positional encoding, and similarity calculation and feature weighting. Its main idea is to adaptively generate feature weights based on feature similarity, making the differences between important and unimportant parts of the features more significant. The general formula for the attention mechanism is shown below:

[0070] Q,K,V=α(F+P),β(F+P),γ(F)

[0071]

[0072] Where F represents the input features of the attention mechanism, α, β, and γ represent linearized layers or multilayer perceptrons with non-shared weights, P represents the positional encoding features, and Q, K, and V represent the Query, Key, and Value matrices in the attention mechanism, respectively. Then, QK... T Calculate the feature similarity between Q and K, and divide by the scaling factor. After normalization using Softmax, the feature is finally generated by performing a Hadamard product with V.

[0073] The following will combine Figure 3 A detailed explanation of the processes of self-attention and cross-attention:

[0074] The principle behind the self-attention enhancement phase is shown in the following formula:

[0075]

[0076] F r The input features are real point clouds, Q r ,K r V r All are derived from input features F r Obtained through linear transformation, MultiheadAttn represents concatenating the results calculated by multiple attention heads, while LayerNorm represents applying layer normalization to the features. This represents the enhanced features of the output.

[0077] The search point cloud features are used as input to the self-attention mechanism to obtain the reweighted search point cloud features; the template point cloud features are used as input to the self-attention mechanism to obtain the reweighted template point cloud features.

[0078] For pseudo-point cloud features, feature enhancement is achieved using the following formula:

[0079]

[0080] F p The input features are pseudo-point clouds, Q p ,K p V p All are derived from input features F p Obtained through linear transformation, MultiheadAttn represents concatenating the results calculated by multiple attention heads, while LayerNorm represents applying layer normalization to the features. This represents the enhanced features of the output.

[0081] By using the search pseudo-point cloud features as input to the self-attention mechanism, the reweighted search pseudo-point cloud features are obtained; by using the template pseudo-point cloud features as input to the self-attention mechanism, the reweighted template pseudo-point cloud features are obtained.

[0082] In the cross-attention enhancement stage, cross-modal feature enhancement is performed on the output features of the previous stage according to the following formula:

[0083]

[0084]

[0085] in, Self-attention enhancement features Obtained through linear transformation, Self-attention enhancement features Obtained through linear transformation, This represents the final enhanced real-world point cloud features processed by the cross-attention mechanism. This represents the final enhanced pseudo-point cloud features processed by the cross-attention mechanism. Unlike the self-attention mechanism, the cross-attention mechanism enhances the features generated by this branch. As a query, the feature generated by enhancing another branch. Using these as Key and Value, an attention mechanism is then used to calculate feature similarity across different modalities, and feature weights with similar semantic information are enhanced to construct semantic associations between different modalities, ultimately generating semantically enhanced features.

[0086] The reweighted search pseudo-point cloud and search point cloud features are used as input features for cross-attention to obtain the final semantically enhanced search pseudo-point cloud and real point cloud features; the reweighted template pseudo-point cloud and template point cloud features are used as input features for cross-attention to obtain the final semantically enhanced template pseudo-point cloud and real point cloud features.

[0087] Step 3: Combine self-attention and cross-attention to achieve multimodal feature interaction and enhancement; including:

[0088] Step 3.1: Construct long-distance dependencies within a modality through a self-attention mechanism, learn the importance of different features, and weight the features accordingly;

[0089] Step 3.2: Construct semantic connections between different modalities through cross-attention, so that information from different modalities can strengthen the features of the same semantic object, thereby generating more robust multimodal semantic features;

[0090] Multimodal similarity fusion: leveraging semantic enhancement features from point clouds and pseudo-point clouds Generate geometric similarity Sgeo ∈R W×L×D Texture similarity S tex ∈R N×D Where W and L represent the feature map size, N represents the number of keypoints in the pseudo-point cloud, and D represents the feature dimension. The similarity calculation formula is as follows.

[0091]

[0092] In the formula, Correlation is the cosine similarity function. The final enhanced features represent the search region. The final enhanced feature represents the template region;

[0093] After generating the similarity scores for each modality (real point cloud and pseudo point cloud), the following method is used: Figure 4 The method shown integrates geometric similarity and texture similarity. The specific process is as follows: To ensure the uniformity of different similarity representation forms, this invention first converts the sparse texture similarity in the pseudo-point cloud branch into dense BEV similarity features. That is, the texture similarity features are voxelized, and the similarity features of each voxel are aggregated. Finally, a dense BEV similarity S′ is generated. tex ∈R W×L×D Subsequently, this invention further applies cross-attention to the texture similarity S′. tex and geometric similarity S geo Fusion is performed to generate more robust multimodal similarity features S fus The mathematical expression for similarity fusion is as follows:

[0094] Q geo =S geo W Q ,K tex =S′ tex W K V tex =S′ tex W V

[0095]

[0096] S′ fus =MLP(S fus )+S fus

[0097] Where MLP represents a multilayer perceptron, Q geo ,K tex V tex This represents the Query generated from geometric similarity, and the Key and Value generated from texture similarity. This step encodes the geometric and texture cues of the target into multimodal similarity features S′.fus Therefore, it is beneficial to improve the robustness of the tracker and overcome failure problems in complex environments.

[0098] Step 4: Cross-modal modeling capability based on attention mechanism fuses image texture similarity and point cloud geometric similarity; including:

[0099] Step 4.1: Apply a pixel-wise cosine similarity calculation function to the semantic enhancement features of the search point cloud and the template point cloud to generate a geometric similarity S. geo ∈R W×L×D In the formula, W×L×D represents the tensor S geo The dimension;

[0100] Step 4.2: Apply a point-by-point cosine similarity calculation function to the semantic enhancement features of the search pseudo-point cloud and the template pseudo-point cloud, where the texture similarity S... tex ∈R N×D In the formula, N×D represents the tensor S tex The dimension;

[0101] Step 4.3: Using a cross-attention mechanism, geometric similarity and texture similarity are fused to generate a more robust multimodal similarity feature S′. fus ;

[0102] Target location prediction: After obtaining the fused similarity features in step 4, the similarity feature S′ is... fus The target location is classified by using branches of a convolutional neural network, and the target's relative offset position and orientation are regressed.

[0103] Classification branch: S′ fus Each feature center is used as a key point The classification branch needs to predict the score for each keypoint and output a heatmap. The keypoint with the highest classification score in the heatmap indicates that the target is most likely to be present at that location. To construct the constraints of the loss function, this invention generates the heatmap ground truth Y∈[0,1] using a Gaussian kernel. W×H×1 The Gaussian kernel formula used is as follows:

[0104]

[0105] Where δ p The adaptive standard deviation of the target size is used, and then Focal Loss is used as the loss function to optimize the network parameters. The loss function is shown in the following formula:

[0106]

[0107] Where α and β are the hyperparameters of Focal Loss, and N is the similarity feature S′. fusNumber of key points, L h This represents the value of the FocalLoss loss function;

[0108] The regression branch primarily predicts the target's three-dimensional properties. It includes positional deviation. high and orientation The positional deviation is used to correct the keypoint positions to obtain more accurate target localization results. Subsequently, this invention uses the L1 loss function to calculate the loss values ​​of the target's three-dimensional attributes:

[0109]

[0110]

[0111] Among them, L offset L represents the value of the positional error loss function. z L represents the value of the height loss function. θ Represents the value of the loss function, {x offset y offset} represents the position of the sample target used to train the model, z represents the height of the sample target used to train the model, and θ represents the orientation of the sample target used to train the model.

[0112] Finally, the final loss value L is obtained by weighted summation.

[0113] L=λ h L h +λ offset L offset +λ z L z +λ θ L θ

[0114] Where L is the total loss function, λ h λ represents the coefficient of the Focal Loss function. offset λ is the coefficient of the deviation loss function. z λ is the coefficient of the height loss function. θ The coefficients are those of the loss function.

[0115] Step 5: Based on the multimodal similarity feature S′ fus It employs a multi-layer convolutional structure to predict the spatial location and orientation of a target; including:

[0116] Step 5.1: Extract target cues hidden in multimodal similarity features through a multi-layer convolutional structure to generate feature maps;

[0117] Step 5.2: Predict the target's confidence level and corresponding spatial attributes at each location in the feature map. The location with the highest confidence level is the target's undetermined location. The spatial attributes are the target's position deviation and orientation, which are used to correct the undetermined location.

[0118] Step 5.3: Use the corrected undetermined position as the prediction result for the tracking target.

[0119] This invention proposes a 3D single-target tracking method based on multimodal information fusion, which solves the problem that single-modal trackers are prone to confusing geometrically similar objects, thus obtaining more robust tracking results. The main improvements are as follows:

[0120] (1) Multimodal data spatial alignment: By using point cloud depth and camera intrinsic and extrinsic parameters, pixels are mapped to the spatial coordinate system where the point cloud is located, generating pseudo point clouds with texture features. This helps reduce the difficulty of fusing multimodal features in subsequent networks and improves the utilization rate of image features.

[0121] (2) Multimodal dual-stream feature extraction network, which efficiently extracts multimodal features through different backbone networks. The point cloud branch generates dense BEV geometric features, which helps to alleviate the impact of point cloud sparsity. The pseudo point cloud branch can rely on a small number of points to represent the local texture information of the target and background, which helps to reduce the computational cost of subsequent networks.

[0122] (3) Multimodal information interaction and enhancement mechanism: First, the important features within the modality are adaptively enhanced by the self-attention mechanism. Second, the semantic association of the multimodality is constructed through the cross-attention mechanism, thereby strengthening the features of the same semantic object.

[0123] (4) Multimodal similarity fusion: Adaptive fusion of geometric and texture similarity is achieved through attention mechanism to generate coded target geometric and texture similarity features, which are then used for final target regression to improve the robustness of the tracker in complex scenarios such as sparse point clouds or interference (a large number of structurally similar objects).

Claims

1. A three-dimensional single-target tracking method based on multimodal information fusion, characterized in that, include: Step 1: Spatial alignment of images and point cloud data acquired by different sensors; Step 2: Construct a dual-stream feature extraction network based on deep learning methods to enable parallel extraction of high-dimensional semantic features of pseudo-point clouds and point clouds from different network branches; Step 3: Combine self-attention and cross-attention to achieve interaction and enhancement of multimodal features; Step 4: Based on the semantic enhancement features generated in Step 3, calculate the texture similarity of the image and the geometric similarity of the point cloud, and fuse the two types of similarity features with cross attention to generate multimodal similarity. Step 5: Based on multimodal similarity features It uses a multi-layer convolutional structure to predict the spatial location and orientation of the target; Step 2 includes: Step 2.1: Extract texture features from the pseudo-point cloud; Step 2.1 includes: Step 2.1.1: For the pseudo-point cloud obtained in Step 1, process the template pseudo-point cloud. and search for pseudo-point clouds The farthest point sampling algorithm was used to downsample the point cloud, and Q points were obtained as key points respectively. Step 2.1.2: For the template pseudo-point cloud and search pseudo-point cloud obtained in step 2.1.1, KNN clustering is performed on the points within a radius R centered on Q key points, and then MLP network is used to aggregate the features of the clustered points into the key points. Step 2.1.3: Repeat the application of farthest point sampling, KNN clustering, and MLP network to reduce the number of keypoints to N, and the output includes... Template point set of points and search point set ; each of them From a three-dimensional coordinate vector and a representation of local texture information of an object dimensional descriptor Composition, that is ; Step 2.2: Extract the geometric features of the real point cloud; Step 2.2 includes: Step 2.2.1: For the real point cloud, convert the template point cloud... and search point cloud Voxelization is performed to convert it into a dense voxel representation; Step 2.2.2: For the template point cloud and search point cloud Three-dimensional sparse convolutional networks are used to extract the geometric features of point clouds in voxels to generate template point clouds. Three-dimensional voxel features Search point cloud Three-dimensional voxel features In the formula, Indicates the dimension of the corresponding tensor; Step 2.2.3: Targeting 3D voxel features , The height channel and feature channel are merged separately to output denser BEV features. .

2. The three-dimensional single-target tracking method based on multimodal information fusion according to claim 1, characterized in that, Step 1 includes: Step 1.1: Project the point cloud onto the image plane based on the camera's intrinsic and extrinsic parameters; Step 1.2: Assign the point cloud depth to the corresponding pixels; Step 1.3: Backproject the pixels into three-dimensional space to generate a pseudo point cloud that has image texture information and is aligned with the original point cloud coordinate system.

3. The three-dimensional single-target tracking method based on multimodal information fusion according to claim 1, characterized in that, Step 3 includes: Step 3.1: Construct long-distance dependencies within a modality through a self-attention mechanism, learn the importance of different features, and weight the features accordingly; Step 3.2: Construct semantic connections between different modalities through cross-attention, so that information from different modalities can strengthen the features of the same semantic object, thereby generating more robust multimodal semantic features.

4. The three-dimensional single-target tracking method based on multimodal information fusion according to claim 1, characterized in that, Step 4 includes: Step 4.1: Generate geometric similarity by applying a pixel-wise cosine similarity calculation function to the semantic enhancement features of the search point cloud and the template point cloud. In the formula, Tensor The dimension; Step 4.2: Apply a point-by-point cosine similarity calculation function to the semantic enhancement features of the search pseudo-point cloud and the template pseudo-point cloud to generate texture similarity. In the formula, Tensor The dimension; Step 4.3: Fusing geometric similarity and texture similarity using a cross-attention mechanism to generate more robust multimodal similarity features. .

5. The three-dimensional single-target tracking method based on multimodal information fusion according to claim 1, characterized in that, Step 5 includes: Step 5.1: Extract target cues hidden in multimodal similarity features through a multi-layer convolutional structure to generate feature maps; Step 5.2: Predict the target's confidence level and corresponding spatial attributes at each location in the feature map. The location with the highest confidence level is the target's undetermined location. The spatial attributes are the target's position deviation and orientation, which are used to correct the undetermined location. Step 5.3: Use the corrected undetermined position as the prediction result for the tracking target.

Images