Three-dimensional target detection and track line identification method based on multi-modal fusion
By employing deep learning and multimodal fusion methods, feature extraction and clustering are performed on image and point cloud data to generate a unified query set. This addresses the issue of insufficient detection capabilities of a single sensor in complex environments, enabling high-precision track line recognition and target detection.
Patent Information
- Application Number
- CN202511787253.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-01
- Publication Date
- 2026-02-06
AI Technical Summary
Existing single-sensor-based orbital environment perception methods perform poorly in 3D geometric reconstruction, long-distance detection, and low-light conditions. Furthermore, multimodal fusion methods fail to fully utilize the complementary information between different modes, resulting in low detection accuracy.
Deep learning methods are used to extract features from image data and point cloud data to generate image features and point cloud features. Clustering and decoding are performed using image query and point cloud query to generate a unified query set. Multimodal fusion is performed using self-attention mechanism and center clustering method to generate the final target detection and trajectory line recognition results.
It significantly improves detection accuracy and robustness in complex environments, enhances the accuracy and consistency of long-distance target detection, reduces computational load, and avoids coordinate transformation errors.
Smart Images
Figure CN121482770A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of intelligent rail transit and computer vision, and relates to a method for three-dimensional target detection and track line recognition based on multimodal fusion. Background Technology
[0002] With the rapid development of intelligent rail transit technology, the requirements for the accuracy and robustness of track environment perception are becoming increasingly stringent. However, perception methods based on single sensors have significant shortcomings. While camera sensors can provide rich texture information and semantic details, their performance degrades significantly in 3D geometric reconstruction, long-range detection, and low-light conditions such as nighttime. LiDAR sensors, although providing accurate 3D spatial information, are insufficient in semantic understanding and fine-grained target recognition, and are prone to false detections in long-range and sparse point cloud scenarios. Furthermore, most 3D track line detection methods rely on vision-centric approaches, i.e., monocular solutions. As vision-centric solutions, these monocular 3D methods inevitably fail to capture the complexity of real-world driving scenarios when encountering adverse weather and lighting conditions. In contrast, LiDAR excels in spatial localization and 3D structure perception, complementing the capabilities of passive sensor cameras.
[0003] Existing multimodal 3D detection methods generally employ simple feature stitching or post-fusion strategies, failing to fully utilize the complementary information between different modalities and exhibiting low accuracy in long-distance detection. Therefore, this paper proposes a 3D target detection and trajectory line recognition method based on multimodal fusion. Summary of the Invention
[0004] The purpose of this invention is to provide a three-dimensional target detection and track line recognition method based on multimodal fusion, which is applicable to long-distance target detection and track line recognition scenarios in rail transit.
[0005] The technical solution of the present invention:
[0006] A method for 3D target detection and track line recognition based on multimodal fusion is proposed. Deep learning is used to extract features from image data and point cloud data to generate image features, point cloud voxel features, and point cloud features, respectively. For target detection, the extracted image features and point cloud voxel features are used for 2D and 3D detection. The image features, point cloud voxel features, and detection results are then input into a point cloud query generator and an image query generator to generate image queries and point cloud queries, respectively. The generated image queries and point cloud queries are clustered to generate new queries. These new queries are input into the decoding layer, and finally, the target detection result is output through the detection head. For track line recognition, the extracted image features and point cloud features are used to generate image queries and point cloud queries through a unified query generator. The two types of queries are then clustered to generate new track queries. The new track queries are input into the decoding layer, and finally, the track line detection result is output through the detection head. The specific steps are as follows:
[0007] S1 acquires image data from the camera and point cloud data from the LiDAR for target detection and trajectory line recognition;
[0008] S2, extract image features from image data using a ResNet34 network. Point cloud features are extracted from point cloud data using the PillarNet34 network. Simultaneously, voxel features are extracted from the point cloud data using the FSDv2 network. and 3D pre-detection results ;
[0009] S3, using image features from S2 2D pre-detection results are generated using an image 2D detector. ;
[0010] S4, image features and 2D pre-detection results Input image query generator to get image query ,in This indicates the part containing the query content. and This represents the 3D sampling location and its corresponding probability. The voxel features... and 3D pre-detection results Input point cloud query generator to get point cloud query ,in This indicates the part containing the query content. Indicates the location part;
[0011] S5. Since the different query formats in step S4 are different, based on the self-attention mechanism, the query content part is retained, and the position part is converted into a coherent representation. Generate a unified format query, expressed as , where m represents img or pc. Specifically, the positional coding method (PE) and the uncertainty-aware positional coding method (U-PE) are used respectively. and Create a location code using the following expression:
[0012] (1)
[0013] SinPos represents the sinusoidal position code.
[0014] In U-PE, we first will Transformed into basic positional encoding Then, the probability is controlled through gating operations. Injected into middle:
[0015] (2)
[0016] Flat() represents the flattening operation, ⊙ represents element-wise multiplication, and σ represents the Sigmoid function.
[0017] S6 proposes a centroid clustering method to generate a unified query set for object detection. Specifically, image queries are used as cluster centers C, and queries from point clouds are used to... Each query in C is assigned to the nearest cluster center, as shown in the following expression:
[0018] (3)
[0019] S7, after obtaining the unified query set Next, a decoder with six decoding layers is constructed, all using the same internal architecture. Specifically, each decoding layer consists of self-attention, cross-attention, and query update. The decoder outputs the updated... ,Will The data is input into the detection head to obtain 3D target detection results;
[0020] S8, using image features from S2 and point cloud features The images are then input into a unified query generator to obtain the query results. Point cloud query ;
[0021] S9, query based on image And point cloud query Referring to clustering methods in object detection, a new query is obtained. ;
[0022] S10, after obtaining the unified query set Next, two identical decoding layers are stacked as a decoder. Each layer predicts a 3D reference point and its 3D offset by querying, fully utilizing the geometric properties of 3D space to generate 3D deformation points. After two layers of decoding iterations, the updated query... The data is input into the detection head, which ultimately outputs the track line detection results.
[0023] The beneficial effects of this invention are as follows: This invention fully utilizes the complementary advantages of image processing and LiDAR, significantly improving detection accuracy and robustness in complex environments; it adopts a sparse query approach, avoiding dense feature mapping, and achieving long-distance target detection while reducing computational load; by fusing orbit queries under different modalities, it effectively integrates the orbital geometric structure information of image processing and LiDAR, improving detection accuracy and consistency; and it performs multimodal fusion and geometric constraints in a unified coordinate system, avoiding coordinate transformation errors and improving the overall system performance. Attached Figure Description
[0024] Figure 1 This is a schematic diagram of the target detection query generation process in an embodiment of the present invention;
[0025] Figure 2 This is a schematic diagram of the overall track detection process in an embodiment of the present invention;
[0026] Figure 3 This is a schematic diagram of the central clustering technology in an embodiment of the present invention. Detailed Implementation
[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, other embodiments obtained by those skilled in the art without creative effort are all within the scope of protection of the present invention.
[0028] The purpose of this invention is to provide a three-dimensional target detection and track line recognition method based on multimodal fusion, which is applicable to long-distance target detection and track line recognition scenarios in rail transit.
[0029] This method uses deep learning to extract features from image data and point cloud data, generating image features, point cloud voxel features, and point cloud features respectively. For object detection, the extracted image features and point cloud voxel features are used for 2D and 3D detection. The features and detection results are then input into a point cloud query generator and an image query generator to generate image queries and point cloud queries. The generated image queries and point cloud queries are clustered to generate new queries. These new queries are input into the decoding layer, and finally, the target detection result is output through the detection head. For track line recognition, the extracted image features and point cloud features are used to generate image queries and point cloud queries respectively through a unified query generator. The two types of queries are then clustered to generate new track queries. The new track queries are input into the decoding layer, and finally, the track line detection result is output through the detection head. The specific steps are as follows:
[0030] S1, acquire image data from the camera and point cloud data from the LiDAR;
[0031] S2, For image data, a ResNet34 network is used for feature extraction, and multi-scale features are generated through FPN. Point cloud features are extracted from point cloud data using the PillarNet34 network. Simultaneously, voxel features are extracted from the point cloud data using the FSDv2 network. and 3D pre-detection results ;
[0032] S3, after obtaining the image features, 2D object detection is performed on the image features using a 2D detection head to generate 2D bounding boxes. The 2D detection head adopts a two-stage detector architecture: first, candidate boxes are generated through the RPN; then, classification and regression are performed through the ROI Head to generate the 2D bounding boxes. It includes information such as the target's category, location, and confidence level;
[0033] S4, image features and 2D pre-detection results Input image query generator to get image query Voxel features and 3D pre-detection results Input point cloud query generator to get point cloud query ;
[0034] S5. To perform query fusion, the generated query format is standardized. Specifically, position codes are created for each channel using both the Position Encoding (PE) method and the Uncertainty Awareness Position Encoding (U-PE) method, as shown in the following expressions:
[0035] (1)
[0036] SinPos represents the sinusoidal position code.
[0037] In U-PE, we first will Transformed into basic positional encoding Then, the probability is controlled through gating operations. Injected into middle:
[0038] (2)
[0039] Where Flat() represents the flattening operation, ⊙ represents element-wise multiplication, and σ represents the Sigmoid function. The unified format query expression is as follows: , where m represents img or pc;
[0040] S6 uses a centroid clustering method to generate a unified query set for object detection. Specifically, image queries are used as cluster centers C, and queries from point clouds are used to... Each query in C is assigned to the nearest cluster center, as shown in the following expression:
[0041] (3)
[0042] S7, after obtaining the unified query set Next, a decoder with six decoding layers is constructed, all using the same internal architecture. Specifically, each decoding layer consists of self-attention, cross-attention, and query update. Finally, the updated... The input is fed into the detection head to obtain 3D target detection results, which include 7 bounding box parameters (x, y, z, l, w, h, rot).
[0043] S8, using image features from S2 and point cloud features The images are then input into a unified query generator to obtain the query results. And point cloud query Taking point cloud query generation as an example, the specific expression is as follows:
[0044] (4)
[0045] in, It's the sigmoid function; Concat represents the concatenation operation. These are two channel spatial positioning features;
[0046] S9, query based on image And point cloud query Referring to the clustering method in object detection methods, a new query is obtained. ;
[0047] S10, will The input is fed into the decoder, which consists of two identical decoding layers stacked together. Each layer predicts a 3D reference point and its 3D offset by querying, making full use of the geometric properties of 3D space to generate 3D deformation points. After two layers of decoding iteration, the updated query is input into the detection head, and the final output is the trajectory line detection result.
[0048] Based on the above-described preferred embodiments of the present invention, and through the foregoing description, those skilled in the art can make various changes and modifications without departing from the technical spirit of the present invention. The technical scope of the present invention is not limited to the contents of the specification, but must be determined according to the scope of the claims.
Claims
1. A method for three-dimensional target detection and trajectory line recognition based on multimodal fusion, characterized in that, Deep learning methods are used to extract features from image data and point cloud data to generate image features, point cloud voxel features, and point cloud features, respectively. For object detection, the extracted image features and point cloud voxel features are used for 2D and 3D detection to obtain detection results. Then, the image features, point cloud voxel features, and detection results are input into a point cloud query generator and an image query generator to generate image queries and point cloud queries. The generated image queries and point cloud queries are clustered to generate new queries, which are then input into the decoding layer. Finally, the detection head outputs the object detection results. For track line recognition, the extracted image features and point cloud features are used to generate image queries and point cloud queries, respectively, through a track line query generator. Then, the image queries and point cloud queries are clustered to generate new track queries, which are then input into the decoding layer. Finally, the detection head outputs the track line detection results.
2. The method for three-dimensional target detection and trajectory line recognition based on multimodal fusion according to claim 1, characterized in that, The specific steps are as follows: S1 acquires image data from the camera and point cloud data from the LiDAR for target detection and trajectory line recognition; S2, Image features are extracted from the image data using a ResNet34 network. Point cloud data was processed using a PillarNet34 network to extract point cloud features. Simultaneously, point cloud voxel features are extracted from the point cloud data using the FSDv2 network. and 3D pre-detection results ; S3, using the image features from step S2 2D pre-detection results are generated using an image 2D detector. ; S4, image features and 2D pre-detection results Input image query generator to get image query ,in This indicates the part containing the query content. and Representing the positional part and the corresponding probability; voxel features and 3D pre-detection results Input point cloud query generator to get point cloud query ,in This indicates the part containing the query content. Indicates the location part; S5. Since the different query formats in step S4 are different, based on the self-attention mechanism, the query content part is retained, and the position part is converted into a coherent representation. Generate a unified format query, expressed as Where m represents img or pc; specifically, the positional coding method PE and the uncertainty-aware positional coding method U-PE are used for image querying. And point cloud query Create a location code using the following expression: （1）； Wherein, SinPos represents the sinusoidal position code; In the uncertainty-aware location coding method, the location part is first... Transform into positional encoding Then, the probability is controlled through gating operations. Inject into position encoding In detail, as follows: （2）； Where Flat() represents the flattening operation, ⊙ represents element-wise multiplication, and σ represents the Sigmoid function; S6 proposes a centroid clustering method to generate a unified query set for object detection. Query by image As cluster center C, and in the future, point cloud queries Each query in the query is assigned to the nearest cluster center in cluster center C, as shown in the following expression: (3) ； S7, after obtaining the unified query set Next, a decoder with six decoding layers is constructed, each using the same internal architecture; the structure of each decoding layer is: self-attention, cross-attention, and query update; the decoder outputs the updated... ,Will The data is input into the detection head to obtain 3D target detection results; S8, using the image features from step S2 and point cloud features The images are obtained by inputting them into the orbital line query generator. And point cloud query ; S9, query based on image And point cloud query Based on object detection clustering methods, image queries are performed. Cluster centers are used for calculation to obtain new queries. ; S10, upon receiving a new query Next, two identical decoding layers are stacked as a decoder; each layer predicts a 3D reference point and its 3D offset by querying, making full use of the geometric properties of 3D space to generate 3D deformation points; after two layers of decoding iteration, a new query... The data is input into the detection head, which ultimately outputs the track line detection results.