A three-dimensional track target detection method based on visible light and radar point cloud data

Abstract

The application discloses a three-dimensional track target detection method based on visible light and radar point cloud data, comprising the following steps: preparing visible light images and radar point cloud data sets of a track transportation scene; constructing a fusion detection model, wherein the model comprises a lightweight image feature extraction module, a three-dimensional multi-scale voxel feature aggregation module, a two-dimensional bird's eye view feature generation module and a differential fusion module; training the data sets, wherein the image branch extracts multi-scale image features and is subjected to enhancement processing, the point cloud branch extracts voxel features, aggregates multi-scale voxel features and converts the multi-scale voxel features into two-dimensional bird's eye view features, and then the unified multi-modal features are generated by the differential fusion module; and adopting a joint multi-task loss function to optimize the model parameters; while ensuring real-time inference speed, the precision and robustness of three-dimensional target detection in a track scene are significantly improved, visible light texture information and radar geometric structure are effectively fused, and the method is suitable for intelligent track transportation applications such as train automatic driving and level crossing early warning.

Description

Technical Field

[0001] This invention relates to the field of target detection technology, and in particular to a three-dimensional orbital target detection method based on visible light and radar point cloud data. Background Technology

[0002] 3D target detection is a crucial research area in intelligent sensing, aiming to accurately acquire the 3D position, size, and contour of targets such as vehicles and obstacles in rail transit scenarios. With the development of multimodal sensing technology, 3D target detection is widely applied in scenarios such as automatic train operation, level crossing warnings, and infrastructure inspection, playing a key role in the intelligent upgrading and safe operation of rail transit systems. In challenging environments such as complex weather, low light, and strong backlight, rail transit target 3D detection technology that integrates visible light and radar data plays an irreplaceable role. However, due to the large differences in target scale, frequent mutual occlusion, and complex background structures in rail transit scenarios, as well as the susceptibility of visible light sensors to lighting conditions and the sparseness and significant noise of radar point cloud data, achieving robust and accurate 3D detection still faces many unique challenges. While traditional single-sensor-based 3D detection methods each have their advantages, they still have shortcomings in generalization, real-time performance, and accuracy when dealing with the complex dynamic environment of rail transit. Therefore, comprehensively utilizing the rich texture information of visible light images and the accurate ranging capability of radar data to construct an efficient and robust 3D detection system for rail transit targets is a key technology that urgently needs to be broken through, and it is also a current research focus in the field of intelligent transportation.

[0003] Chinese patent publication number "CN121259351A", entitled "A Method, Apparatus, Computer Equipment, and Storage Medium for Track Traffic Safety Target Detection Based on Image Data Fusion", describes a method that first acquires three modal images of the track area: visible light, infrared thermal imaging, and synthetic aperture radar. These images are then preprocessed, including registration and scaling, to eliminate spatial differences between the different modalities. Next, a weighted fusion of the feature maps from the three modalities is performed using an attention mechanism, integrating their respective advantageous features. The fused feature map is then input into a target detection model trained based on a rotated bounding box regression mechanism to obtain rotated bounding boxes of targets in the track area. Finally, the intersection-union ratio (IUU) of overlapping rotated bounding boxes is calculated, and redundant detection boxes are removed to determine the final target detection result.

[0004] However, this method still faces significant challenges in target detection within the track area. First, it relies on simultaneous multi-sensor acquisition and high-precision registration. In the complex and ever-changing environment along the track, differences in sensor installation and environmental disturbances can lead to decreased accuracy in image synchronization and registration, affecting subsequent fusion results. Second, while the designed cross-modal attention fusion mechanism can dynamically weight different features, there is still room for optimization in how to more precisely model the reliability of each modality under different environmental conditions to achieve more robust adaptive fusion. Furthermore, this method primarily focuses on the detection process; how to continuously optimize the model to cope with novel and rare obstacles constantly appearing in the track environment—that is, the model's long-tail learning and continuous adaptation capabilities—has not been explored in depth. Therefore, designing a track safety target detection method that can effectively improve the robustness of multi-source image alignment, achieve more intelligent modal fusion, and enhance the model's adaptability in open environments is a problem that urgently needs further resolution. Summary of the Invention

[0005] The technical solution of this invention to solve the above-mentioned technical problems is to provide a three-dimensional orbital target detection method based on visible light and radar point cloud data, comprising the following steps:

[0006] S1, Prepare the dataset: Prepare three publicly available target detection datasets for rail scene: Dataset 1 is the Rail3D dataset; Dataset 2 is the RailSem19 dataset; Dataset 3 is the Osdar23 dataset, which are visible light images and radar point cloud data of rail traffic scenes used for training, validation and testing.

[0007] S2, Constructing the detection model: Constructing a detection model based on the fusion of visible light and radar data, the model includes:

[0008] A lightweight image feature extraction module for extracting multi-scale image features from visible light images;

[0009] A 3D multi-scale voxel feature aggregation module is used to extract and aggregate multi-scale voxel features from radar point clouds.

[0010] The 2D bird's-eye view feature generation module is used to convert the aggregated voxel features into 2D bird's-eye view features;

[0011] The differential fusion module is used to fuse the multi-scale image features with the two-dimensional bird's-eye view features;

[0012] S3, Training the network model: Using the data prepared in step S1, the detection model constructed in step S2 is trained. In the visible light image data processing branch, multi-scale image features are extracted through the lightweight image feature extraction module and processed sequentially through the adaptive spatial feature fusion and attention enhancement module to output enhanced visible light features.

[0013] In the radar point cloud data processing branch, the three-dimensional features of the point cloud are extracted by the voxel feature encoding module, and then processed by the three-dimensional multi-scale voxel feature aggregation module and the two-dimensional bird's-eye view feature generation module to output the enhanced radar features.

[0014] The enhanced visible light features and the enhanced radar features are input into the differential fusion module to generate unified multimodal features for training;

[0015] S4, Selecting the loss function: Optimize the model parameters using a joint multi-task loss function.

[0016] Furthermore, the lightweight image feature extraction module adopts a two-stage design, including a low-level feature extraction part and a main feature extraction part;

[0017] The underlying feature extraction part uses a cascaded and parallel structure to construct a large receptive field feature;

[0018] The main feature extraction part adopts a grouping and heterogeneous processing mechanism, and achieves lightweight feature extraction through a combination of parallel topology and grouped convolution.

[0019] Furthermore, the three-dimensional multi-scale voxel feature aggregation module uses the center point of the non-empty voxel in the final downsampling layer as the query point, and retrieves and fuses voxel features from shallower layers within a spherical neighborhood of its preset radius, thereby constructing a multi-scale spatial feature encoding mechanism that integrates geometric details and semantic information.

[0020] The feature encoding mechanism uses set abstract compression operation to sample the spherical neighborhood, and applies MLP and pooling operation to aggregate features in each grouped region. Local multi-scale features are obtained by setting different spherical radii, and multi-scale geometric features are generated through hierarchical sampling and aggregation operations.

[0021] Furthermore, the two-dimensional bird's-eye view feature generation module utilizes fully sparse height compression technology to convert three-dimensional voxel features into a two-dimensional bird's-eye view feature map rich in spatial semantic information.

[0022] The two-dimensional bird's-eye view feature generation module uses the three-dimensional features generated by the data upscaling method as a supervision signal to guide the learning process of the two-dimensional feature map;

[0023] The fully sparse high compression technology is SHC technology, and the data dimensionality enhancement method is implemented based on the BEVFusion model.

[0024] Furthermore, the SHC operation is as follows:

[0025] ;

[0026] in, This represents the two-dimensional bird's-eye view features obtained after the voxel features output from the l-th layer of 3D-MVFA are processed by SHC. Features of a 3D bird's-eye view It is a two-dimensional position coordinate on the bird's-eye view plane. Indicates a two-dimensional position Above, the set of all non-empty voxels along the Z-axis. It is a certain non-empty voxel eigenvectors; Indicates to The eigenvectors of all voxels in the set are summed and aggregated to generate the two-dimensional position. final eigenvalues ;

[0027] The 2D bird's-eye view feature generation module is designed with an auxiliary branch to perform top-down fusion of SHC features generated from 3D features of different layers, so as to fuse the high-resolution detail information of the shallow layer and the strong semantic information of the deep layer; to fuse feature maps of different scales to the same resolution, and to downsample the feature maps of larger scales. ;in, These are features from a sparse bird's-eye view with a higher resolution layer. These are features that are downsampled and aligned with the resolution of the current layer l.

[0028] The aligned features are then fused with the original SHC features of the current layer. The sparse two-dimensional features described above are then converted into dense feature maps and further feature extraction is performed.

[0029] ;

[0030] in, This represents the fused set of multi-scale sparse features. The operation transforms sparse BEV features into a dense two-dimensional feature tensor. Features for the final two-dimensional bird's-eye view.

[0031] Furthermore, the differential fusion module includes a preliminary fusion module and a core fusion module;

[0032] The preliminary fusion module is used to concatenate the visible light features and the radar features along the channel dimension, and generate preliminary fused features through convolution dimensionality reduction;

[0033] The core fusion module is used to generate a spatial attention weight mask through convolution and the Sigmoid function, and adaptively add the weighted visible light features to the radar features by adding the residuals.

[0034] Furthermore, the joint multi-task loss function mentioned in step S4 includes: classification loss, orientation loss, centroid shift loss, and focus loss;

[0035] The classification loss is used to evaluate the degree of class matching between the predicted bounding box and the ground truth bounding box;

[0036] The orientation loss is used to optimize the angular accuracy of the predicted bounding box;

[0037] The centroid offset loss is used to minimize the deviation between the predicted target center point and the true center point;

[0038] The focus loss is used to adjust the weights of easy and difficult samples to alleviate the class imbalance problem.

[0039] Furthermore, it also includes the following steps:

[0040] S5, Fine-tuning the model: Evaluate model performance using multi-dimensional evaluation metrics; Fine-tune the trained model using validation data to optimize model parameters;

[0041] S6, Save Model: Solidify the fine-tuned network parameters to obtain the final three-dimensional orbital target detection model.

[0042] Furthermore, the datasets mentioned in step S1 include: the Rail3D dataset, the RailSem19 dataset, and the Osdar23 dataset.

[0043] Furthermore, the joint multi-task loss function includes a classification loss L. cls Directional loss L dir Centroid shift loss L cen and focus loss L focal ;

[0044] centroid shift loss L cen The formula is shown below:

[0045] ;

[0046] ;

[0047] ;

[0048] Where Cbox and Cgt represent the centroids of the predicted bounding box and the ground truth bounding box, respectively. L2 norm loss; This is a normalization factor used to normalize the loss based on the bounding box size;

[0049] Focus loss L focal As stated in the formula.

[0050] ;

[0051] ;

[0052] ;

[0053] in, It is the probability of predicting it as positive. It is a weighting parameter that balances positive and negative samples, and it has different values ​​for positive and negative samples; It is the focusing parameter for adjusting the loss of hard-to-classify samples; and Set them to the default values ​​of 0.25 and 2 respectively;

[0054] The total loss is as described in the formula:

[0055] ;

[0056] in The number of positive anchor points is represented by λ1, λ2, λ3, and λ4, which are hyperparameters used to balance the weights of the loss components.

[0057] Compared with existing technologies, this invention provides a three-dimensional orbital target detection method based on visible light and radar point cloud data, which has the following beneficial effects:

[0058] (1) This invention designs a three-dimensional detection model for rail transit targets based on visible light and radar data. The model is based on a multi-level feature optimization and fusion mechanism, which realizes the complementary fusion of visible light and radar data, effectively improving the target detection accuracy and robustness in rail transit scenarios.

[0059] (2) The present invention designs a lightweight image feature extraction module to efficiently capture and optimize visual semantic information, laying the foundation for subsequent image feature extraction.

[0060] (3) The present invention designs a three-dimensional multi-scale voxel feature aggregation module and a two-dimensional bird's-eye view feature generation module to enhance the representation of radar data point cloud and construct highly discriminative view features.

[0061] (4) The present invention designs a differential fusion module to efficiently fuse the processed visible light image features and two-dimensional radar features, realizing deep complementarity and adaptive weighted fusion between visible light data features and radar data features, which significantly enhances the model's ability to integrate multi-source heterogeneous information.

[0062] (5) The detection model proposed in this invention has shown good results on the test data in Rail3D, RailSem19 and Osdar23 datasets, and the quantitative evaluation indicators have been improved. This shows that the detection method proposed in this invention can achieve stable three-dimensional detection in rail transit scenarios. Attached Figure Description

[0063] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0064] Figure 1 This is a flowchart illustrating the steps of the three-dimensional orbital target detection method based on visible light and radar point cloud data described in this invention.

[0065] Figure 2 This is a schematic diagram of the three-dimensional detection model for rail transit targets constructed in this invention;

[0066] Figure 3 This is a structural diagram of the lightweight image feature extraction module described in this invention;

[0067] Figure 4 This is a structural diagram of the three-dimensional multi-scale voxel feature aggregation module and the two-dimensional bird's-eye view feature generation module described in this invention;

[0068] Figure 5 This is a structural diagram of the differential fusion module described in this invention;

[0069] Figure 6 This is a qualitative comparison of the three-dimensional detection methods for rail transit targets based on visible light and radar data described in this paper. Detailed Implementation

[0070] This invention proposes a three-dimensional track target detection method based on visible light and radar point cloud data. The aim is to design a track vehicle safety target detection method that can effectively improve the robustness of multi-source image alignment, achieve more intelligent modal fusion, and enhance the model's adaptability in open environments.

[0071] The three-dimensional orbital target detection method based on visible light and radar point cloud data proposed in this invention will be described below in specific embodiments:

[0072] Example 1:

[0073] A method for detecting three-dimensional orbital targets based on visible light and radar point cloud data includes the following steps:

[0074] S1, Dataset Preparation: Rail3D contains 3058 images with annotations for "obstacles" and "pedestrians". RailSem19 contains 8500 images taken from the perspective of rail vehicles, showcasing various scenes in rail transportation. osdar23 contains 1534 images and 204,091 labeled objects.

[0075] S2, Constructing the detection model: Constructing a detection model based on the fusion of visible light and radar data, the model includes:

[0076] A lightweight image feature extraction module for extracting multi-scale image features from visible light images;

[0077] A 3D multi-scale voxel feature aggregation module is used to extract and aggregate multi-scale voxel features from radar point clouds.

[0078] The 2D bird's-eye view feature generation module is used to convert the aggregated voxel features into 2D bird's-eye view features;

[0079] The differential fusion module is used to fuse the multi-scale image features with the two-dimensional bird's-eye view features;

[0080] S3, Training the network model: Using the data prepared in step S1, the detection model constructed in step S2 is trained. In the visible light image data processing branch, multi-scale image features are extracted through the lightweight image feature extraction module and processed sequentially through the adaptive spatial feature fusion and attention enhancement module to output enhanced visible light features.

[0081] In the radar point cloud data processing branch, the three-dimensional features of the point cloud are extracted by the voxel feature encoding module, and then processed by the three-dimensional multi-scale voxel feature aggregation module and the two-dimensional bird's-eye view feature generation module to output the enhanced radar features.

[0082] The enhanced visible light features and the enhanced radar features are input into the differential fusion module to generate unified multimodal features for training;

[0083] S4, Selecting the loss function: Optimize the model parameters using a joint multi-task loss function.

[0084] Furthermore, the lightweight image feature extraction module adopts a two-stage design, including a low-level feature extraction part and a main feature extraction part;

[0085] The underlying feature extraction part uses a cascaded and parallel structure to construct a large receptive field feature;

[0086] The main feature extraction part adopts a grouping and heterogeneous processing mechanism, and achieves lightweight feature extraction through a combination of parallel topology and grouped convolution.

[0087] Furthermore, the three-dimensional multi-scale voxel feature aggregation module uses the center point of the non-empty voxel in the final downsampling layer as the query point, and retrieves and fuses voxel features from shallower layers within a spherical neighborhood of its preset radius, thereby constructing a multi-scale spatial feature encoding mechanism that integrates geometric details and semantic information.

[0088] The feature encoding mechanism uses set abstract compression operation to sample the spherical neighborhood, and applies MLP and pooling operation to aggregate features in each grouped region. Local multi-scale features are obtained by setting different spherical radii, and multi-scale geometric features are generated through hierarchical sampling and aggregation operations.

[0089] Furthermore, the two-dimensional bird's-eye view feature generation module utilizes fully sparse height compression technology to convert three-dimensional voxel features into a two-dimensional bird's-eye view feature map rich in spatial semantic information.

[0090] The two-dimensional bird's-eye view feature generation module uses the three-dimensional features generated by the data upscaling method as a supervision signal to guide the learning process of the two-dimensional feature map;

[0091] The fully sparse high compression technology is SHC technology, and the data dimensionality enhancement method is implemented based on the BEVFusion model.

[0092] Furthermore, the SHC operation is as follows:

[0093] ;

[0094] in, This represents the two-dimensional bird's-eye view features obtained after the voxel features output from the l-th layer of 3D-MVFA are processed by SHC. Features of a 3D bird's-eye view It is a two-dimensional position coordinate on the bird's-eye view plane. Indicates a two-dimensional position Above, the set of all non-empty voxels along the Z-axis. It is a certain non-empty voxel eigenvectors; Indicates to The eigenvectors of all voxels in the set are summed and aggregated to generate the two-dimensional position. final eigenvalues ;

[0095] The 2D bird's-eye view feature generation module is designed with an auxiliary branch to perform top-down fusion of SHC features generated from 3D features of different layers, so as to fuse the high-resolution detail information of the shallow layer and the strong semantic information of the deep layer; to fuse feature maps of different scales to the same resolution, and to downsample the feature maps of larger scales. ;in, These are features from a sparse bird's-eye view with a higher resolution layer. These are features that are downsampled and aligned with the resolution of the current layer l.

[0096] The aligned features are then fused with the original SHC features of the current layer. The sparse two-dimensional features described above are then converted into dense feature maps and further feature extraction is performed.

[0097] ;

[0098] in, This represents the fused set of multi-scale sparse features. The operation transforms sparse BEV features into a dense two-dimensional feature tensor. Features for the final two-dimensional bird's-eye view.

[0099] Furthermore, the differential fusion module includes a preliminary fusion module and a core fusion module;

[0100] The preliminary fusion module is used to concatenate the visible light features and the radar features along the channel dimension, and generate preliminary fused features through convolution dimensionality reduction;

[0101] The core fusion module is used to generate a spatial attention weight mask through convolution and the Sigmoid function, and adaptively add the weighted visible light features to the radar features by adding the residuals.

[0102] Furthermore, the joint multi-task loss function mentioned in step S4 includes: classification loss, orientation loss, centroid shift loss, and focus loss;

[0103] The classification loss is used to evaluate the degree of class matching between the predicted bounding box and the ground truth bounding box;

[0104] The orientation loss is used to optimize the angular accuracy of the predicted bounding box;

[0105] The centroid offset loss is used to minimize the deviation between the predicted target center point and the true center point;

[0106] The focus loss is used to adjust the weights of easy and difficult samples to alleviate the class imbalance problem.

[0107] Furthermore, it also includes the following steps:

[0108] S5, Fine-tuning the model: Evaluate model performance using multi-dimensional evaluation metrics; Fine-tune the trained model using validation data to optimize model parameters;

[0109] S6, Save Model: Solidify the fine-tuned network parameters to obtain the final three-dimensional orbital target detection model.

[0110] Furthermore, the datasets mentioned in step S1 include: the Rail3D dataset, the RailSem19 dataset, and the Osdar23 dataset.

[0111] Furthermore, the joint multi-task loss function includes a classification loss L. cls Directional loss L dir Centroid shift loss L cen and focus loss L focal ;

[0112] centroid shift loss L cen The formula is shown below:

[0113] ;

[0114] ;

[0115] ;

[0116] Where Cbox and Cgt represent the centroids of the predicted bounding box and the ground truth bounding box, respectively. L2 norm loss; This is a normalization factor used to normalize the loss based on the bounding box size;

[0117] Focus loss L focal As stated in the formula.

[0118] ;

[0119] ;

[0120] ;

[0121] in, It is the probability of predicting it as positive. It is a weighting parameter that balances positive and negative samples, and it has different values ​​for positive and negative samples; It is the focusing parameter for adjusting the loss of hard-to-classify samples; and Set them to the default values ​​of 0.25 and 2 respectively;

[0122] The total loss is as described in the formula:

[0123] ;

[0124] in The number of positive anchor points is represented by λ1, λ2, λ3, and λ4, which are hyperparameters used to balance the weights of the loss components.

[0125] Example 2:

[0126] A method for detecting 3D orbital targets based on visible light and radar point cloud data, such as Figure 1 As shown, it includes the following steps:

[0127] S1, Prepare the dataset: Prepare three publicly available orbital scene object detection datasets: Dataset 1 is the Rail3D dataset; Dataset 2 is the RailSem19 dataset; Dataset 3 is the Osdar23 dataset. Based on Dataset 1, Dataset 2, and Dataset 3, construct a hybrid orbital scene object detection dataset RailData, which contains a total of 13,092 orbital scene object images. These images are allocated in an 8:1:1 ratio for training, testing, and fine-tuning of the network model.

[0128] S2, Construct a 3D detection model for rail transit targets: The detection model is as follows... Figure 2 As shown, the model introduces a dual-modal processing design: In the visible light data feature processing branch, a lightweight image feature extraction module is designed to efficiently extract multi-scale features from the input image. This feature is then fused using adaptive spatial feature fusion technology to generate optimized features with adaptive weighted fusion, thereby improving the efficiency of multi-scale feature fusion. To further enhance feature representation capabilities, an attention enhancement module is introduced to enhance the optimized feature in both channel and spatial dimensions, ultimately outputting enhanced visible light data features. In the radar feature processing branch, an end-to-end feature dimensionality reduction extraction process is adopted. First, the original point cloud 3D features are extracted using a voxel feature encoding module. Then, a 3D multi-scale voxel feature aggregation module is used to generate aggregated features, mitigating the loss of quantization information during voxelization and enhancing multi-scale target features. Addressing the problem of uneven distribution of non-empty voxel space in rail transit scenarios, a 3D bird's-eye view supervised feature based on bird's-eye view fusion is introduced, and a 2D bird's-eye view feature generation module is used to fuse multi-level spatial features, ultimately outputting strongly representative 2D radar features. In the multimodal fusion stage, the enhanced visible light features are fused with the strongly representative two-dimensional radar features through a differential fusion module. An attention mechanism is used to strengthen the complementary information between the two modalities, generating a unified multimodal feature representation. Finally, a 3D bounding box regression strategy based on the detector and center point is applied to achieve efficient and accurate prediction of the target's 3D detection box.

[0129] S2, Construct a lightweight image feature extraction module (LIFE) in the 3D detection model of rail transit targets, such as Figure 3 As shown, (1) shows the low-level feature extraction part of LIFE. Taking Kernel=3 as an example, R-Conv1 consists of three 3×3 convolutions, which reduces the computational load while obtaining a large receptive field of the convolution kernel. R-Conv2 adopts a dual-branch structure to further obtain a larger receptive field, and then performs feature fusion through 1×1 convolution. (2) is the main feature extraction part of LIFE, where n represents the number of residual blocks. Unlike ResNet50, LIFE divides the input channels into two groups. One group is directly processed by various custom convolutions, and the other group is processed by simple convolution. This avoids learning duplicate information by different convolutions and maintains detection accuracy while reducing computational load. By combining grouped convolutions and asymmetric convolutions, the network achieves lightweighting while efficiently extracting and fusing features with the help of parallel topology. In order to verify the effectiveness of the LIFE module, a two-dimensional comparison experiment was conducted on the RailData dataset using LIFE and existing image feature extraction modules. The comparison results are shown in Table 1.

[0130] Table 1:

[0131]

[0132] S2, in constructing a 3D detection model for rail transit targets, the 3D multi-scale voxel feature aggregation module is as follows: Figure 5 As shown, the 3D multi-scale voxel feature aggregation module aims to achieve cross-scale feature enhancement in the final downsampling layer of the network. Using the center point of the non-empty voxel in the final downsampling layer as the query point, the module retrieves and fuses voxel features from shallower layers within a spherical neighborhood of its preset radius, thereby constructing a multi-scale spatial feature encoding mechanism capable of fusing information from low-level geometric details to high-level semantics. As described in the formula, the features of the non-empty voxels are encoded into 3D sparse feature volumes with downsampling scales of 1x, 2x, 4x, and 8x.

[0133] ;

[0134] in, This represents the set of voxel locations that constitute the neighborhood of the convolution kernel. Let be the weight matrix and b be the bias matrix.

[0135] like Figure 4 As shown, the 3D multi-scale voxel feature aggregation module employs set abstraction (SA) operations to sample the spherical neighborhood around each query point. Then, MLP and pooling operations are applied within each independent grouping region to aggregate features. By setting different sphere radii, local multi-scale features at the center point are obtained. Then, hierarchical sampling and aggregation operations are performed to summarize the multi-scale geometric features. The specific operations are as follows.

[0136] The final downsampling layer of the 3D voxel convolutional network is designated as the aggregation layer, and the center of the voxel points in this layer is used as the query point Q of the SA. The spatial coordinates of the query point... It is calculated based on the voxel index and the actual voxel dimension, where 'a' represents the number of query points. The set of voxel feature vectors for this layer is denoted as... The spatial coordinates of the voxel in the k-th downsampling layer are: The corresponding voxel feature vector set is , where N k It is the number of non-empty voxels in the k-th layer.

[0137] For each query point For the radius neighborhood The non-empty voxels of the k-th inner layer are sampled in groups to obtain the feature vector set of the grouped voxels, which can be represented as:

[0138] ;

[0139] in Indicates in the neighborhood Collect the 16 most recent voxels and their local relative coordinates. With features The process of connecting them.

[0140] Subsequently, the PointNet module is used to generate local aggregate features for the query points.

[0141] ;

[0142] in This represents a three-layer MLP network that encodes features. Operations This represents max pooling along the channel, aggregating multi-scale features to the query point. Furthermore, two different scales are used at the k-th layer. This increases the acceptor field of local variables.

[0143] Finally, all k downsampling layers are used as query points. The generated aggregated features are concatenated to obtain their enhanced multi-scale voxel features. .

[0144] ;

[0145] S2, In constructing the 3D detection model for rail transit targets, the 2D bird's-eye view feature generation module is as follows: Figure 4As shown, the core idea of ​​the 2D bird's-eye view feature generation module is to efficiently convert 3D voxel features into 2D bird's-eye view feature maps rich in multi-level spatial semantic information using a fully sparse high-compression technique. The 2D bird's-eye view feature generation module uses 3D bird's-eye view features generated by the BEV Fusion data upscaling method as a supervision signal to guide the learning process of the 2D feature map, thereby significantly reducing computational overhead while maintaining rich geometric and semantic information. The SHC operation, as described in the formula, maps non-empty voxels onto the XY plane and accumulates features at the same locations.

[0146] ;

[0147] in This represents the two-dimensional bird's-eye view features obtained after the voxel features output from the l-th layer of 3D-MVFA are processed by SHC. Features of a 3D bird's-eye view It is a two-dimensional position coordinate on the bird's-eye view plane. Indicates a two-dimensional position Above, the set of all non-empty voxels along the Z-axis. It is a certain non-empty voxel The feature vectors of . Operation Indicates to The eigenvectors of all voxels in the set are summed and aggregated to generate the two-dimensional position. final eigenvalues .

[0148] Furthermore, inspired by FPN, the 2D bird's-eye view feature generation module incorporates an auxiliary branch that performs top-down fusion of SHC features generated from different layers of 3D features, thereby integrating high-resolution detail information from shallow layers with strong semantic information from deeper layers. To fuse feature maps of different scales to the same resolution, downsampling of the larger-scale feature maps is necessary.

[0149] ;

[0150] in, These are features from a sparse bird's-eye view with a higher resolution layer. It is a feature that is downsampled and aligned with the resolution of the current layer l.

[0151] The aligned features are then fused with the original SHC features of the current layer.

[0152] ;

[0153] In order to provide the subsequent differential fusion module with multi-modal data fusion, the above sparse two-dimensional features need to be converted into dense feature maps and further feature extraction needs to be performed.

[0154] ;

[0155] in, This represents the fused set of multi-scale sparse features. The operation transforms sparse BEV features into a dense two-dimensional feature tensor. Features for the final two-dimensional bird's-eye view.

[0156] S2, In constructing the 3D detection model of rail transit targets, the differential fusion module is as follows: Figure 5 As shown, the differential fusion module first concatenates the visible light data features obtained after final processing with the two-dimensional radar features, and then reduces the dimensionality through convolution fl(.) to generate preliminary fused features. .

[0157] ;

[0158] Here, [.;.] represents a concatenation operation along the channel dimension, and fl(.) is a 1x1 convolution.

[0159] Then The process is divided into two paths. One path generates a spatial attention weight mask through convolution and the sigmoid function. In the core fusion step, residuals are added using a formula, meaning that the weighted image features are adaptively added to the 2D bird's-eye view features, thereby effectively enhancing detail perception while preserving reliable spatial structure.

[0160] ;

[0161] Here, gl(.) represents the convolution operation, used to extract features from the fusion process. An unnormalized attention map is generated.

[0162] The sigmoid activation function compresses the values ​​of the attention map to between 0 and 1, generating a spatially adaptive weight mask. The closer the weight is to 1, the more reliable and important the image feature corresponding to that location is.

[0163] S3, Training the detection network model: Input the training data from dataset 1, dataset 2 and dataset 3 prepared in step S1 into the target 3D detection model constructed in step S2 for training. The above network architecture and training process were implemented using Python 3.10 and PyTorch 2.3, and efficient parallel computing was performed on NVIDIA Titan Xp GPU. During training, the Adam optimizer was selected, and the parameters λ1=0.5, λ2=0.6, λ3=0.8 and λ4=0.9 were set to accelerate the convergence process and reduce oscillations. All training data and test data were uniformly adjusted to 640×480 resolution. The number of training iterations was set to 200, and the batch size was set to 16. In terms of optimization method selection, the MultiStepLR

[49] learning rate decay strategy was adopted. This strategy dynamically adjusts the learning rate, provides a larger learning rate in the early stage of training to accelerate convergence, and then gradually decreases it to avoid overfitting. The learning rate was set to 5×10 −4 The attenuation factor is set to 0.1.

[0164] S4. Select a suitable loss function and determine the optimal evaluation index of the present invention: The loss function is a joint multi-task loss function that jointly optimizes the three-dimensional detection model of rail transit targets. This includes classification loss L... cls and directional loss L dir Furthermore, due to the adoption of a center-point regression-based 3D bounding box detection strategy, an additional centroid offset loss L is introduced. cen This approach directly optimizes the center position of the predicted bounding box, making it closer to the centroid of the ground truth bounding box. Specifically, it performs cluster analysis on the point cloud data within the bounding box, calculates the offset between the centroids of the predicted and ground truth boxes, and uses this offset to guide model training. cen The specific definition is as described in the formula.

[0165] ;

[0166] ;

[0167] ;

[0168] Where Cbox and Cgt represent the centroids of the predicted bounding box and the ground truth bounding box, respectively. The loss is the L2 norm. This is a normalization factor used to normalize the loss based on the bounding box size.

[0169] During centroid calculation, kernel density estimation (KDE) is used to analyze N points within each bounding box. The weight of each point is calculated using a Gaussian kernel function. KDE captures the spatial or feature distribution of the input data, and the choice of the Gaussian kernel ensures the smoothness and robustness of the parameter estimation, and is dynamically adjusted according to the data density.

[0170] Meanwhile, to make the 3D detection model of rail transit targets more focused on small targets that are difficult to detect, this paper adopts the focus loss L... focal As stated in the formula:

[0171] ;

[0172] ;

[0173] ;

[0174] in, It is the probability of predicting it as positive. It is a weighting parameter that balances positive and negative samples, and has different values ​​for positive samples (target) and negative samples (background). It is a focusing parameter that adjusts the loss of hard-to-classify samples. and Set them to the default values ​​of 0.25 and 2 respectively.

[0175] In summary, the total loss is as stated in the formula:

[0176] ;

[0177] in The number of positive anchor points is represented by λ1, λ2, λ3, and λ4, which are hyperparameters used to balance the weights of the loss components.

[0178] S4. In selecting a suitable loss function and determining the optimal evaluation index for this method, the evaluation index adopts commonly used object detection evaluation indexes. These indexes include mean accuracy (mAP), GFLOPs, delay (FPS), model parameter count (Params), mean translation error (ATE), mean scale error (ASE), mean orientation error (AOE), and detection score (NDS). Through the joint analysis of these indexes, the detection performance of the 3D detection model for rail transit targets and its performance in different scenarios can be comprehensively analyzed.

[0179] 3D target detection is similar to 2D target detection, using an average accuracy metric. However, instead of using the IOU criterion, 3D average accuracy is defined by setting a threshold of the 2D center distance *d* on the ground plane. Subsequently, the AP is averaged over a defined set of categories *C* at a set of matching thresholds *D* = {0.5, 1, 2, 4} *m*, as described in the formula:

[0180] ;

[0181] True positive TP metrics are employed to further quantify the effectiveness of the 3D detection model for rail transit targets. These TP metrics use a center distance of d = 2 meters during the matching process. These TP metrics are designed as positive scalars, providing a comprehensive assessment of the frame performance. TP errors include: ATE calculated using the 2D Euclidean center distance; ASE defined by the 3D IOU after alignment of orientation and position, with a value of (1 − IOU); and AOE, the minimum yaw angle difference between the prediction and the ground truth. For each TP metric, the average TP is calculated across all classes, as described in the formula.

[0182] ;

[0183] Considering the limitations of 3D mAP in fully capturing the complexity of 3D object detection tasks, the NDS metric is introduced. NDS provides a more comprehensive evaluation by assessing different aspects of detection performance separately, as described in the formula.

[0184] ;

[0185] In addition, FPS is used to evaluate the computational speed of the network, and the specific formula is as described in the formula.

[0186] ;

[0187] Where t is the time required to process a single frame, in seconds.

[0188] S5, Fine-tuning the model: The model is retrained and fine-tuned using the validation data from datasets 1, 2, and 3 prepared in step S1. The learning rate is set to 0.005, and a total of 200 iterations are performed while keeping other parameters unchanged, to further improve the performance of the detection network.

[0189] S6, Save the model: After training is completed in step S3, solidify the fine-tuned network parameters. In step S5, fine-tune the model to determine the final three-dimensional detection network model of the rail transit target.

[0190] To verify the effectiveness of the proposed 3D detection model for rail transit targets, a 3D comparative experiment was conducted on the RailData dataset using the detection model and existing single-modal detection models based on radar point cloud data and multi-modal detection models based on multi-source data fusion. The comparison results are shown in Table 2.

[0191] Table 2:

[0192]

[0193] The detection categories are shown in Table 3.

[0194] Table 3:

[0195]

[0196] Qualitative comparison, for example Figure 6 As shown in the figure, the method proposed in this invention improves the performance of 3D target detection in rail transit, achieving an mAP of 63.24%. 3D With 58.38% NDS, compared to the baseline model BEVFusion, mAP 3D The performance of NDS and radar data was improved by 12.35% and 7.01%, respectively. Furthermore, compared to single-radar data methods, the method proposed in this invention demonstrates superior detection performance. Although some metrics are not optimal, the method maintains considerable computational efficiency, which is particularly important for edge computing devices.

[0197] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for detecting three-dimensional orbital targets based on visible light and radar point cloud data, characterized in that, Includes the following steps: S1, Prepare the dataset: Prepare visible light images and radar point cloud data of rail transit scenes for training, validation and testing; S2, Constructing the detection model: Constructing a detection model based on the fusion of visible light and radar data, the model includes: A lightweight image feature extraction module for extracting multi-scale image features from visible light images; A 3D multi-scale voxel feature aggregation module is used to extract and aggregate multi-scale voxel features from radar point clouds. The 2D bird's-eye view feature generation module is used to convert the aggregated voxel features into 2D bird's-eye view features; The differential fusion module is used to fuse the multi-scale image features with the two-dimensional bird's-eye view features; S3, Training the network model: Using the data prepared in step S1, the detection model constructed in step S2 is trained. In the visible light image data processing branch, multi-scale image features are extracted through the lightweight image feature extraction module and processed sequentially through the adaptive spatial feature fusion and attention enhancement module to output enhanced visible light features. In the radar point cloud data processing branch, the three-dimensional features of the point cloud are extracted by the voxel feature encoding module, and then processed by the three-dimensional multi-scale voxel feature aggregation module and the two-dimensional bird's-eye view feature generation module to output the enhanced radar features. The enhanced visible light features and the enhanced radar features are input into the differential fusion module to generate unified multimodal features for training; S4, Selecting the loss function: Optimize the model parameters using a joint multi-task loss function.

2. The method according to claim 1, characterized in that, The lightweight image feature extraction module adopts a two-stage design, including a low-level feature extraction part and a main feature extraction part; The underlying feature extraction part uses a cascaded and parallel structure to construct features with a large receptive field. The main feature extraction part adopts a grouping and heterogeneous processing mechanism, and achieves lightweight feature extraction through a combination of parallel topology and grouped convolution.

3. The method according to claim 1, characterized in that, The three-dimensional multi-scale voxel feature aggregation module uses the center point of the non-empty voxel in the final downsampling layer as the query point, and retrieves and fuses voxel features from shallower layers in a spherical neighborhood with a preset radius to construct a multi-scale spatial feature encoding mechanism that integrates geometric details and semantic information. The feature encoding mechanism uses set abstract compression operation to sample the spherical neighborhood, and applies MLP and pooling operation to aggregate features in each grouped region. It sets different spherical radii to obtain local multi-scale features, and then generates multi-scale geometric features through hierarchical sampling and aggregation operations.

4. The method according to claim 1, characterized in that, The two-dimensional bird's-eye view feature generation module uses fully sparse height compression technology to convert three-dimensional voxel features into two-dimensional bird's-eye view feature maps rich in spatial semantic information. The two-dimensional bird's-eye view feature generation module uses the three-dimensional features generated by the data upscaling method as a supervision signal to guide the learning process of the two-dimensional feature map; The fully sparse high compression technology is SHC technology, and the data dimensionality enhancement method is implemented based on the BEVFusion model.

5. The method according to claim 4, characterized in that, The SHC operation is as follows: ； in, This represents the two-dimensional bird's-eye view features obtained after the voxel features output from the l-th layer of 3D-MVFA are processed by SHC. Features of a 3D bird's-eye view It is a two-dimensional position coordinate on the bird's-eye view plane. Indicates a two-dimensional position Above, the set of all non-empty voxels along the Z-axis. It is a certain non-empty voxel eigenvectors; Indicates to The eigenvectors of all voxels in the set are summed and aggregated to generate the two-dimensional position. final eigenvalues ; The 2D bird's-eye view feature generation module is designed with an auxiliary branch to perform top-down fusion of SHC features generated from 3D features of different layers, so as to fuse the high-resolution detail information of the shallow layer and the strong semantic information of the deep layer; to fuse feature maps of different scales to the same resolution, and to downsample the feature maps of larger scales. ;in, These are features from a sparse bird's-eye view with a higher resolution layer. These are features that are downsampled and aligned with the resolution of the current layer l. The aligned features are then fused with the original SHC features of the current layer. The sparse two-dimensional features described above are then converted into dense feature maps and further feature extraction is performed. ； in, This represents the fused set of multi-scale sparse features. The operation transforms sparse BEV features into a dense two-dimensional feature tensor. Features for the final two-dimensional bird's-eye view.

6. The method according to claim 1, characterized in that, The differential fusion module includes a preliminary fusion module and a core fusion module; The preliminary fusion module is used to concatenate the visible light features and the radar features along the channel dimension, and generate preliminary fused features through convolution dimensionality reduction; The core fusion module is used to generate a spatial attention weight mask through convolution and the Sigmoid function, and adaptively add the weighted visible light features to the radar features by adding the residuals.

7. The method according to claim 1, characterized in that, The joint multi-task loss function mentioned in step S4 includes: classification loss, orientation loss, centroid shift loss, and focus loss; The classification loss is used to evaluate the degree of class matching between the predicted bounding box and the ground truth bounding box; The orientation loss is used to optimize the angular accuracy of the predicted bounding box; The centroid offset loss is used to minimize the deviation between the predicted target center point and the true center point; The focus loss is used to adjust the weights of easy and difficult samples to alleviate the class imbalance problem.

8. The method according to claim 1, characterized in that, It also includes the following steps: S5, Fine-tuning the model: Evaluate model performance using multi-dimensional evaluation metrics; Fine-tune the trained model using validation data to optimize model parameters; S6, Save Model: Solidify the fine-tuned network parameters to obtain the final three-dimensional orbital target detection model.

9. The method according to claim 1, characterized in that, The datasets mentioned in step S1 include: Rail3D dataset, RailSem19 dataset, and Osdar23 dataset.

10. The method according to claim 7, characterized in that, centroid shift loss L cen The formula is shown below: ； ； ； Where Cbox and Cgt represent the centroids of the predicted bounding box and the ground truth bounding box, respectively. L2 norm loss; This is a normalization factor used to normalize the loss based on the bounding box size; Focus loss L focal As stated in the formula; ； ； ； in, It is the probability of predicting it as positive. It is a weighting parameter that balances positive and negative samples, and it has different values ​​for positive and negative samples; It is the focusing parameter for adjusting the loss of hard-to-classify samples; and Set them to the default values ​​of 0.25 and 2 respectively; The total loss is as described in the formula: ； in The number of positive anchor points is represented by λ1, λ2, λ3, and λ4, which are hyperparameters used to balance the weights of the loss components.

Images