Railway tunnel foreign matter detection method based on radar point cloud and RGB image fusion

Abstract

The application discloses a railway tunnel foreign matter detection method based on radar point cloud and RGB image fusion, comprising the following steps: acquiring laser radar point cloud and an RGB image in a railway tunnel; acquiring an RGB image to judge a railway tunnel foreign matter abnormal state by using a GAN (Generative Adversarial Network) model; if the foreign matter abnormal state occurs, projecting the obtained laser radar point cloud to an image plane to obtain a dense depth map; obtaining a data slice approximately in a square shape through sliding window processing; performing data feature extraction, fusion and foreign matter target detection on the data slice of the RGB image and the dense depth map by using a fusion target detection network based on EfficientNet-FPN, and outputting foreign matter target information detection results containing a target category, a position, a confidence and a distance. The railway tunnel foreign matter detection method based on radar point cloud and RGB image fusion has high precision, high efficiency and high reliability in railway tunnel foreign matter detection.

Description

Technical Field

[0001] This invention belongs to the field of railway tunnel safety detection technology, specifically relating to a method for detecting foreign objects in railway tunnels based on the fusion of radar point clouds and RGB images. Background Technology

[0002] Railway tunnels, as crucial transportation infrastructure, play a vital role in protecting people's lives and property. However, the presence of foreign objects within tunnels can pose a serious threat to train operation and even cause accidents. Currently, foreign object detection methods in railway tunnels mainly include technologies based on video image analysis and lidar point cloud analysis. Video image analysis methods rely on visible light images, are easily affected by factors such as lighting conditions and weather changes, and have low accuracy in low-light environments. While lidar point cloud analysis methods can provide high-precision three-dimensional spatial information, they have limitations in processing speed and real-time performance, and their ability to identify foreign objects in complex environments needs improvement.

[0003] Furthermore, existing foreign object detection systems also have certain problems in data acquisition and processing. For example, data acquisition equipment may not be able to acquire environmental information inside the tunnel at high frequency and in real time, affecting the real-time performance and accuracy of the detection. Regarding data processing, how to effectively integrate data from multiple sensors to improve the accuracy and robustness of foreign object detection is a key challenge. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method for detecting foreign objects in railway tunnels based on the fusion of radar point cloud and RGB image, so as to achieve high precision, high efficiency and high reliability in detecting foreign objects in railway tunnels.

[0005] To achieve the above objectives, the present invention is implemented using the following technical solution:

[0006] In a first aspect, the present invention provides a method for detecting foreign objects in railway tunnels based on the fusion of radar point clouds and RGB images, comprising the following steps:

[0007] LiDAR point cloud and RGB image of railway tunnel were acquired in real time using LiDAR and high-definition camera at the same data acquisition frequency;

[0008] The GAN (Generated Adversarial Network) model, which utilizes a fused graph attention network (GAT) and an autoencoder (AE), is used to acquire data feature relationships in RGB images. Through adversarial training, data reconstruction, and error reconstruction, the system determines the occurrence of foreign object anomalies in railway tunnels. If a foreign object anomaly occurs:

[0009] The obtained lidar point cloud is projected onto the image plane to obtain a sparse depth map, and then a dense depth map is obtained through depth completion processing.

[0010] The obtained dense depth map and RGB image are processed by a sliding window to obtain approximately square data slices;

[0011] The EfficientNet-FPN-based fusion target detection network is used to extract, fuse, and detect foreign objects in data slices of RGB images and dense depth maps, and output foreign object detection results including target category, location, confidence level, and distance.

[0012] Furthermore, before projecting the obtained lidar point cloud onto the image plane, the lidar point cloud and RGB image acquired by the lidar and high-definition camera are transformed from different coordinate data to the same coordinate system at the same data acquisition frequency. The transformation relationship from the lidar point cloud coordinate system to the image pixel coordinate system is expressed as follows:

[0013]

[0014] In the formula, (u,v) represents the coordinates of the point cloud in the pixel coordinate system; R is a 3x3 rotation matrix, representing the rotation from the lidar point cloud coordinate system to the camera coordinate system; T LC This is a 4x4 matrix representing the extrinsic parameter matrix from the lidar point cloud coordinate system to the camera coordinate system, containing rotation and translation information; t X t Y t Z Let X be the translation vector, representing the position of the origin of the lidar coordinate system in the camera coordinate system; L ,Y L Z L ) represents the coordinates of a point in the lidar coordinate system; K is the 3x3 intrinsic parameter matrix of the camera, representing the projection from the camera coordinate system to the pixel coordinate system; f X The focal length of the camera in the X direction, expressed in pixels, is f. Y c represents the camera's focal length in the Y direction, in pixels. X c represents the coordinates of the camera's principal point in the X direction, in pixels. Y The coordinates of the camera's principal point in the Y direction are represented in pixels. The camera's intrinsic parameters and projection matrix are provided by the equipment used in the experiment. The coordinates (u,v) of the laser point cloud in the image plane can be calculated based on the transformation formula.

[0015] Furthermore, the LiDAR point cloud is projected onto the image plane to obtain a sparse depth map, which is then filled in to obtain a dense depth map. This dense depth map is then optimized using the depth map target method, as expressed by the formula:

[0016] f(I,D sparse ) = D dense ;

[0017]

[0018] In the formula, I represents the input image, Dsparse represents the sparse depth map, and Ddense represents the dense depth map generated by the depth completion algorithm; f(I,D) represents the dense depth map predicted by the model. sparse ) represents the true dense depth map, and the optimization objective is to minimize the difference between the model prediction and the true value, which is measured by the Frobenius norm (||·||F).

[0019] Furthermore, in the fusion target detection network based on EfficientNet-FPN, the EfficientNet feature fusion network is used to extract features from the RGB image and the radar point cloud depth map respectively, and the two sets of extracted feature maps are cascaded and fused. Then, the pyramid network FPN is used as the feature detection network to perform pyramid multi-scale processing on the fused feature map, and a residual network is constructed to predict the target using multi-scale features. Non-maximum suppression (NMS) is used for optimization and refinement, and the detection result of foreign object target information including target category, location, confidence level and distance is output.

[0020] Furthermore, the fused object detection network uses EfficientNet-B2 as its base network, with 9.2M parameters and 1.0B floating-point computation. It consists of 5 modules: Module 1 and Module 2 provide basic convolution and pooling operations; Module 3 is used to skip connections between different modules; Module 4 and Module 5 are used to connect feature maps and, together with Module 3, construct the residual network. The combination of different modules constitutes 3 different sub-modules, which are cascaded and fused into the main body of the network.

[0021] Furthermore, in the fused object detection network, Block1 and Block2 of EfficientNet-B2 are used as feature extractors to perform convolution and downsampling on the input RGB image and dense depth map, respectively, to obtain feature maps with consistent depth and scale.

[0022] Feature maps are merged through a feature fusion layer using a 1×1 convolutional kernel. While keeping the feature map scale unchanged, cross-channel information interaction is triggered by increasing non-linearity and reducing the feature map dimension. The fused feature map is obtained by fusing the data features of the two modalities and is fed into the subsequent block of the fusion network for further extraction and downsampling of data features, finally outputting a 13×13 feature map.

[0023] Furthermore, in the fusion target detection network, a Feature Pyramid Network (FPN) structure is adopted to simultaneously learn shallow and deep features on three feature maps with 32x downsampling, 16x downsampling, and 8x downsampling, respectively, in order to respond to and obtain multi-scale target prediction.

[0024] The target location, category, and confidence score are obtained through neural network regression prediction. For each bounding box, the coordinates (x, y), width and height (w, h), and confidence score of the bounding box are predicted.

[0025] Furthermore, the confidence formula is expressed as:

[0026]

[0027] In the formula, This represents the intersection of the predicted bounding box and the ground truth bounding box; if the object exists in the grid, Pr(object) has a value of 1, otherwise it has a value of 0;

[0028] The loss function in the fusion target detection network consists of three parts: coordinate prediction error. coord IoU Error IoU and classification error clas ;

[0029]

[0030] In the formula, λ coord The weights representing coordinate prediction errors; This indicates the indicator function for the j-th object in the i-th grid. These are the prediction results for x, y, w, and h, respectively; λ nobj It is the weight of the IoU error; p(c) is the indicator function for the j-th non-object in the i-th grid; p(c) is the probability of target class c. This is the prediction result.

[0031] Furthermore, the graph attention network (GAT) in the GAN adversarial network model acquires and outputs data features, which can be expressed by the following formula:

[0032]

[0033] In the Generative Adversarial Network (GAN) model, the autoencoder AE takes X as input, maps it to the latent space as variable Z, and maps the latent space variable Z to the original input space of the encoder. The error between the reconstructed X and the original input X is used as the reconstruction error.

[0034] L AE =||X-AE(X)||2;

[0035] In the formula, AE(X)=D(Z), Z=E(X), and ||·||2 represents the L2 norm.

[0036] In a second aspect, the present invention provides an electronic device, comprising:

[0037] processor;

[0038] Memory used to store the processor's executable instructions;

[0039] The processor is configured to execute the instructions to implement the railway tunnel foreign object detection method based on radar point cloud and RGB image fusion as described in any one of the first aspects.

[0040] Compared with the prior art, the beneficial effects achieved by the present invention are as follows:

[0041] This invention provides a method for detecting foreign objects in railway tunnels based on the fusion of radar point clouds and RGB images. It utilizes a LiDAR and a high-definition camera to acquire LiDAR point clouds and RGB images within the railway tunnel in real time at the same data acquisition frequency, ensuring data real-time performance and consistency. A Generative Adversarial Network (GAN) model with a fused graph attention network (GAT) and an autoencoder (AE) is employed to obtain the data feature relationships of the RGB images. Through adversarial training, data reconstruction, and error reconstruction, the detection accuracy of abnormal foreign object states is improved. The LiDAR point cloud is projected onto the image plane to obtain a sparse depth map, which is then processed through depth completion to obtain a dense depth map, enhancing the completeness and accuracy of depth information. Approximately square data slices are obtained through sliding window processing, facilitating subsequent feature extraction and target detection. A fusion target detection network based on EfficientNet-FPN is used to extract features, fuse data from the RGB images and dense depth map slices, and detect foreign objects. The output includes foreign object target information such as target category, location, confidence level, and distance, improving detection accuracy and efficiency. Attached Figure Description

[0042] Figure 1 The flowchart illustrates a method for detecting foreign objects in railway tunnels based on the fusion of radar point clouds and RGB images, as provided in this embodiment of the invention.

[0043] Figures 2-3 This is a diagram of an EfficientNet-B2 architecture provided for an embodiment of the present invention.

[0044] Figure 4 This is a feature fusion layer provided in an embodiment of the present invention.

[0045] Figure 5 This is a diagram of a foreign object detection network architecture provided in an embodiment of the present invention.

[0046] Figure 6 This is a diagram of an autoencoder model provided in an embodiment of the present invention.

[0047] Figure 7 This is a GAN model structure diagram provided in an embodiment of the present invention.

[0048] Figure 8 This is an example diagram of foreign object detection provided in an embodiment of the present invention. Detailed Implementation

[0049] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.

[0050] Example

[0051] like Figure 1 As shown in the figure, this invention provides a method for detecting foreign objects in railway tunnels based on the fusion of radar point clouds and RGB images, including the following steps:

[0052] LiDAR point cloud and RGB image of railway tunnel were acquired in real time using LiDAR and high-definition camera at the same data acquisition frequency;

[0053] The GAN (Generated Adversarial Network) model, which utilizes a fused graph attention network (GAT) and an autoencoder (AE), is used to acquire data feature relationships in RGB images. Through adversarial training, data reconstruction, and error reconstruction, the system determines the occurrence of foreign object anomalies in railway tunnels. If a foreign object anomaly occurs:

[0054] The obtained lidar point cloud is projected onto the image plane to obtain a sparse depth map, and then a dense depth map is obtained through depth completion processing.

[0055] The obtained dense depth map and RGB image are processed by a sliding window to obtain approximately square data slices;

[0056] The EfficientNet-FPN-based fusion target detection network is used to extract, fuse, and detect foreign objects in data slices of RGB images and dense depth maps, and output foreign object detection results including target category, location, confidence level, and distance.

[0057] In this embodiment, before projecting the obtained lidar point cloud onto the image plane, the lidar point cloud and RGB image acquired by the lidar and high-definition camera are transformed from different coordinate data to the same coordinate system at the same data acquisition frequency. The transformation relationship from the lidar point cloud coordinate system to the image pixel coordinate system is expressed as follows:

[0058]

[0059] In the formula, (u,v) represents the coordinates of the point cloud in the pixel coordinate system; R is a 3x3 rotation matrix, representing the rotation from the lidar point cloud coordinate system to the camera coordinate system; T LC This is a 4x4 matrix representing the extrinsic parameter matrix from the lidar point cloud coordinate system to the camera coordinate system, containing rotation and translation information; t X t Y t Z Let X be the translation vector, representing the position of the origin of the lidar coordinate system in the camera coordinate system; L ,Y L Z L ) represents the coordinates of a point in the lidar coordinate system; K is the 3x3 intrinsic parameter matrix of the camera, representing the projection from the camera coordinate system to the pixel coordinate system; f X The focal length of the camera in the X direction, expressed in pixels, is f. Y c represents the camera's focal length in the Y direction, in pixels. X c represents the coordinates of the camera's principal point in the X direction, in pixels. Y The coordinates of the camera's principal point in the Y direction are represented in pixels. The camera's intrinsic parameters and projection matrix are provided by the equipment used in the experiment. The coordinates (u,v) of the laser point cloud in the image plane can be calculated based on the transformation formula.

[0060] The LiDAR point cloud is projected onto the image plane to obtain a sparse depth map, which is then filled in to obtain a dense depth map. Finally, the depth map is optimized for target data. The formula is as follows:

[0061] f(I,D sparse ) = D dense ;

[0062]

[0063] In the formula, I represents the input image, Dsparse represents the sparse depth map, and Ddense represents the dense depth map generated by the depth completion algorithm; f(I, D) represents the dense depth map predicted by the model. sparse ) represents the true dense depth map, and the optimization objective is to minimize the difference between the model prediction and the true value, which is measured by the Frobenius norm (||·||F).

[0064] In this embodiment, the obtained dense depth map and RGB image are subjected to sliding window processing. This means defining a window (usually square or a near-square rectangle) on the image and then sliding this window across the entire image to process the area covered by the window. This can be used to extract fixed-size data slices, which can then be used to train machine learning models, such as deep learning networks. The size of the sliding window can be set according to the model's input requirements. Sliding window processing allows the model to learn features within local regions, which is particularly useful for processing large-size image or video data because it can reduce computation and increase processing speed.

[0065] like Figure 2 and Figure 3 As shown, in the fusion target detection network based on EfficientNet-FPN, the EfficientNet feature fusion network extracts features from the RGB image and the radar point cloud depth map respectively, and the two sets of extracted feature maps are cascaded and fused. Then, the pyramid network FPN is used as the feature detection network to perform pyramid multi-scale processing on the fused feature map, and a residual network is constructed to predict the target using multi-scale features. Non-maximum suppression (NMS) is used for optimization and refinement, and the detection result of foreign object target information including target category, location, confidence level and distance is output.

[0066] The fused object detection network uses EfficientNet-B2 as its base network, with 9.2M parameters and 1.0B floating-point computation. It consists of five modules: Modules 1 and 2 provide basic convolution and pooling operations; Module 3 is used for skip connections between different modules; Modules 4 and 5 are used to connect feature maps and, together with Module 3, construct the residual network. Combinations of these modules form three different sub-modules, which are cascaded and fused into the main network structure, such as... Figure 4 As shown.

[0067] In the fusion target detection network, Block1 and Block2 of EfficientNet-B2 are used as feature extractors to perform convolution and downsampling on the input RGB image and dense depth map respectively to obtain feature maps with consistent depth and scale.

[0068] Feature maps are merged through a feature fusion layer using a 1×1 convolutional kernel. While maintaining the feature map scale, cross-channel information interaction is triggered by increasing non-linearity and reducing feature map dimensionality. This fuses data features from two modalities to obtain a fused feature map, which is then fed into subsequent blocks of the fusion network for further feature extraction and downsampling, ultimately outputting a 13×13 feature map. The network parameter settings are shown in Table 1.

[0069] Table 1 Feature Extraction Network Parameters

[0070]

[0071]

[0072] In the fusion target detection network, a Feature Pyramid Network (FPN) structure is adopted to simultaneously learn shallow and deep features on three feature maps with 32x downsampling, 16x downsampling, and 8x downsampling to respond to and obtain multi-scale target predictions.

[0073] The target location, category, and confidence score are obtained through neural network regression prediction. For each bounding box, the coordinates (x, y), width and height (w, h), and confidence score of the bounding box are predicted.

[0074] The confidence level formula is expressed as follows:

[0075]

[0076] In the formula, This represents the intersection of the predicted bounding box and the ground truth bounding box; if the object exists in the grid, Pr(object) has a value of 1, otherwise it has a value of 0;

[0077] The loss function in the fusion target detection network consists of three parts: coordinate prediction error. coord IoU Error IoU and classification error clas ;

[0078]

[0079] In the formula, λ coord The weights representing coordinate prediction errors; This indicates the indicator function for the j-th object in the i-th grid. These are the prediction results for x, y, w, and h, respectively; λ nobj It is the weight of the IoU error; p(c) is the indicator function for the j-th non-object in the i-th grid; p(c) is the probability of target class c. This is the prediction result.

[0080] In this embodiment, the graph attention network treats the acquired image data as a multi-node graph structure and adaptively allocates the weights of neighboring nodes by proposing the attention relationships between each node within its neighborhood. The input to the graph attention layer is a set of node features. Where N is the number of nodes, and F is the number of features for each node, this layer generates a new set of node features. To obtain sufficient expressive power to transform input features into higher-level features, at least a learnable linear transformation is required. Therefore, in the initial step, a weight matrix W∈R is applied to each node. F′×F A parameterized shared linear transformation is applied. Self-attention is then computed at the nodes, a shared attention mechanism mapping α:R. F′×F →R calculates the attention coefficient:

[0081]

[0082] It represents the importance of node j to node i. In its most general form, this model allows each node to participate in the attention calculation of all other nodes, discarding all structural information. It constructs the graph structure of node i's neighborhood and calculates e. ij Where node j∈N i N i Let j be the neighborhood of node i in the graph. To make the calculated attention coefficients between nodes comparable across different nodes, the softmax function is used to evaluate the attention coefficients e of all nodes j. ij Normalize:

[0083]

[0084] Note that layer α is a single-layer feedforward neural network, composed of weighted vectors. Parameterization is performed and applied to the LeakyReLU nonlinear activation function. After full expansion, the coefficient calculation process for graph attention can be expressed as:

[0085]

[0086] Where →T represents transpose and || represents the join operation. Once α is obtained... ij The normalized attention coefficients are then used to calculate a linear combination of their corresponding features, which serves as the final output feature for each node and is fed into the nonlinear function σ(X), i.e.:

[0087]

[0088] To stabilize the training process, multi-head attention mechanisms will achieve better results. Therefore, multiple independent graph attention mechanisms can be applied, and their output features can be concatenated. The formula for the output features of the multi-head attention mechanism is as follows:

[0089]

[0090] in, Represents the k-th attention mechanism α k The calculated normalized attention coefficient Wk This corresponds to the weight matrix of the linear transformation of the input data. Each node of the final output h' will contain EF' features. If the multi-head attention mechanism is applied to the prediction problem, the output of the multi-head attention mechanism should be averaged in the last layer of the network, and a non-linear activation function σ(X) should be applied to the resulting mean. The calculation method is as follows:

[0091]

[0092] In this embodiment, an autoencoder (AE) is an unsupervised artificial neural network formed by combining an encoder (E) and a decoder (D), such as... Figure 6 As shown. In the encoder part, X is taken as input and mapped to the latent space as variable Z. Conversely, the decoder part maps the latent space variable Z to the original input space of the encoder, denoted as R. The error between the reconstructed result of X and the original input X is called the reconstruction error, which is defined as follows:

[0093] L AE =||X-AE(X)||2;

[0094] In the formula, AE(X)=D(Z), Z=E(X), and ||·||2 represents the L2 norm.

[0095] Autoencoder-based anomaly detection typically uses reconstruction error as the anomaly score; a higher score indicates a greater likelihood of the data being identified as anomalous. These models use only normal data from the target dataset as their training set to learn the characteristics of normal data. While well-trained autoencoder-based anomaly detection models can reconstruct normal data effectively, they struggle to reconstruct anomalous data that was never encountered during training. In actual tunnel structural component anomaly detection, common anomalies are often relatively fixed, including missing fasteners, loose cables, and missing pieces. Therefore, autoencoder-based anomaly detection is better suited for detecting foreign objects in actual tunnels.

[0096] Generative Adversarial Networks (GANs) are unsupervised artificial neural network models capable of judging the authenticity of data. This model consists of two independent networks trained adversarially. One network structure, called the generator (G), aims to generate data that more closely resembles reality, while the other structure, called the discriminator (D), aims to distinguish between the real input data and the data generated by the generator. In other words, the generator's training objective is to increase the likelihood of the discriminator making an incorrect judgment, while the discriminator's training objective is to minimize its own classification error. The network structure of a GAN is shown below. Figure 7 As shown.

[0097] By training on a large number of normal sample data features, and then applying the trained model to actual anomaly detection tasks, the model can identify samples whose features differ significantly from normal samples, thus labeling them as anomalies. Based on the above discussion, graph attention networks can adaptively adjust the weight relationships of points within the neighborhood of a sampling point, thereby better extracting data features. Autoencoders are the most widely used network structure in reconstruction-based anomaly detection models. They extract key features by compressing the input data and then reconstructing the original input through high-dimensional expansion. However, this model is prone to mode collapse and non-convergence during training, while the generative adversarial network (GAN) structure effectively addresses this issue.

[0098] The technical solution of the present invention will be described in conjunction with specific embodiments.

[0099] Experiments were conducted using the KIITI dataset for validation. The experiments consisted of three main parts: First, by comparing the experimental results before and after using the sliding window, the improvement of the sliding window on the detection performance of small targets was verified. Second, the experimental results were used to evaluate the obstacle detection capability of the proposed method under complex lighting conditions, verifying the robustness of the proposed method. Third, the experimental results were used to compare the detection performance of the proposed method and various target detection methods on obstacles in traffic scenes, verifying the improvement of target detection performance by the multimodal data fusion method. The proposed method was compared with Faster-RCNN, YOLOv3, VoxelNet, MV3D, F-PointNet, and AVOD, with the corresponding input data being RGB images, radar point clouds, and fused data, respectively. The detection performance of various methods for Car, Pedestrian, and Cyclist targets in Easy, Moderate, and Hard scenarios is shown in Table 2. The mAP in the table is the average detection accuracy for all targets in Easy, Moderate, and Hard scenarios.

[0100] Table 2 compares the performance of other methods on the KITTI dataset.

[0101]

[0102]

[0103] As can be seen from the data in the table, compared with YOLOv3, VoxelNet, MV3D, and AVOD, the method of this invention (last row) that integrates depth map features and sliding window processing has a comprehensive improvement in accuracy. In achieving [the desired results], [further details are needed].

[0104] While achieving detection accuracy close to that of Faster R-CNN, the detection speed is significantly improved. (Compared with multimodal target-level fusion methods)

[0105] Compared to F-PointNet, it is slightly inferior in detection accuracy, but has a significant improvement in detection speed.

[0106] In summary, the technical solution and method of this invention achieve a balance between detection accuracy and detection speed. Calculations show that the average detection accuracy of the technical solution and method of this invention for Car, Pedestrian, and Cyclist in Easy, Moderate, and Hard scenes is 82.55%, 70.73%, and 57.09%, respectively, with a single-frame calculation time of 0.087s, basically meeting the real-time requirements. Compared to YOLOv3, the fastest single-shot object detection method, the average detection accuracy for Car, Pedestrian, and Cyclist targets is improved by 13.6%, 17.15%, and 9.37% in three scenarios, respectively. Compared to Faster-RCNN based on candidate regions, the detection accuracy is improved by 3.32%, 1.46%, and -5.17%, respectively. Compared to VoxelNet, an object detection method based on laser point clouds, the detection accuracy is improved by 23.15%, 21.68%, and 12.56%, respectively. Compared to AVOD, a detection method based on multimodal data fusion, the detection accuracy is improved by 11.70%, 10.10%, and 2.12%, respectively. The method with added sliding window processing significantly improves the detection accuracy of small targets, but the computation time per frame increases.

[0107] The equipment is fixedly installed on the tunnel sidewall to collect track data in real time for obstacle detection. An obstacle with a cross-sectional area of ​​20x20 cm² is placed 20 meters from the tunnel entrance. Figure 8 Example of an image showing the actual location of an obstacle (left), and the point cloud of the obstacle detected by the algorithm (right).

[0108] This invention utilizes depth information provided by LiDAR as additional features to train a convolutional neural network (CNN). First, a sliding window is used to segment and match the input data to the network. Then, two CNN feature extractors are employed to extract features from the RGB image and the point cloud depth map. An adversarial network model (GAN) is constructed and utilizes a fused graph attention network (GAT) and an autoencoder (AE) to obtain the data feature relationships of the RGB image. Through adversarial training, data reconstruction, and error reconstruction, the abnormal state of foreign objects in railway tunnels is determined. If an abnormal state of foreign objects occurs: the obtained LiDAR point cloud is projected onto the image plane to obtain a sparse depth map, which is then processed through depth completion to obtain a dense depth map. These maps are cascaded to obtain a fused feature map, which is then fed into a target detection network for candidate box position regression and classification. Finally, non-maximum suppression (NMS) is applied to output the detection results, including the target's location, category, confidence level, and distance information.

[0109] Experimental results on the KITTI dataset show that the method of this invention improves the detection robustness under different lighting scenarios by complementing the advantages of multimodal data, and the additional sliding window processing improves the detection effect of small targets.

[0110] Furthermore, embodiments of the present invention provide an electronic device, comprising:

[0111] processor;

[0112] Memory used to store the processor's executable instructions;

[0113] The processor is configured to execute the instructions to implement the railway tunnel foreign object detection method based on radar point cloud and RGB image fusion as described in any one of the first aspects.

[0114] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for detecting foreign objects in railway tunnels based on the fusion of radar point clouds and RGB images, characterized in that, Includes the following steps: LiDAR point cloud and RGB image of railway tunnel were acquired in real time using LiDAR and high-definition camera at the same data acquisition frequency; The GAN (Generated Adversarial Network) model, which utilizes a fused graph attention network (GAT) and an autoencoder (AE), is used to acquire data feature relationships in RGB images. Through adversarial training, data reconstruction, and error reconstruction, the system determines the occurrence of foreign object anomalies in railway tunnels. If a foreign object anomaly occurs: The obtained lidar point cloud is projected onto the image plane to obtain a sparse depth map, and then a dense depth map is obtained through depth completion processing. The obtained dense depth map and RGB image are processed by a sliding window to obtain approximately square data slices; The EfficientNet-FPN-based fusion target detection network is used to extract, fuse, and detect foreign objects in data slices of RGB images and dense depth maps, and output foreign object detection results including target category, location, confidence level, and distance.

2. The method for detecting foreign objects in railway tunnels based on the fusion of radar point clouds and RGB images according to claim 1, characterized in that, Before projecting the obtained lidar point cloud onto the image plane, the lidar point cloud and RGB image acquired by the lidar and high-definition camera are transformed from different coordinate data to the same coordinate system at the same data acquisition frequency. The transformation relationship from the lidar point cloud coordinate system to the image pixel coordinate system is expressed as follows: In the formula, (u,v) represents the coordinates of the point cloud in the pixel coordinate system; R is a 3x3 rotation matrix, representing the rotation from the lidar point cloud coordinate system to the camera coordinate system; T LC It is a 4x4 matrix, representing the extrinsic parameter matrix from the lidar point cloud coordinate system to the camera coordinate system, which includes rotation and translation information; t X t Y t Z Let X be the translation vector, representing the position of the origin of the lidar coordinate system in the camera coordinate system; L ,Y L Z L ) represents the coordinates of a point in the lidar coordinate system; K is the 3x3 intrinsic parameter matrix of the camera, representing the projection from the camera coordinate system to the pixel coordinate system; f X The focal length of the camera in the X direction, expressed in pixels, is f. Y c represents the camera's focal length in the Y direction, in pixels. X c represents the coordinates of the camera's principal point in the X direction, in pixels. Y The coordinates of the camera's principal point in the Y direction are represented in pixels. The camera's intrinsic parameters and projection matrix are provided by the equipment used in the experiment. The coordinates (u,v) of the laser point cloud in the image plane can be calculated based on the transformation formula.

3. The method for detecting foreign objects in railway tunnels based on the fusion of radar point clouds and RGB images according to claim 2, characterized in that, The LiDAR point cloud is projected onto the image plane to obtain a sparse depth map, which is then filled in to obtain a dense depth map. Finally, the depth map is optimized for target data. The formula is as follows: f(I,D sparse )=D dense ; In the formula, I represents the input image, Dsparse represents the sparse depth map, and Ddense represents the dense depth map generated by the depth completion algorithm; f(I, D) represents the dense depth map predicted by the model. sparse ) represents the true dense depth map, and the optimization objective is to minimize the difference between the model prediction and the true value, where the difference is expressed by the Frobenius norm (||·||). F )measure.

4. The method for detecting foreign objects in railway tunnels based on the fusion of radar point clouds and RGB images according to claim 3, characterized in that, In the EfficientNet-FPN-based fusion target detection network, the EfficientNet feature fusion network extracts features from RGB images and radar point cloud depth maps respectively, and the two sets of extracted feature maps are cascaded and fused. Then, the pyramid network FPN is used as the feature detection network to perform pyramid-style multi-scale processing on the fused feature maps, constructs a residual network to predict targets using multi-scale features, optimizes and refines them through non-maximum suppression (NMS), and outputs foreign object target information detection results containing target category, location, confidence level, and distance.

5. The method for detecting foreign objects in railway tunnels based on the fusion of radar point clouds and RGB images according to claim 4, characterized in that, The fusion target detection network uses EfficientNet-B2 as its base network, with 9.2M parameters and 1.0B floating-point computation. It consists of 5 modules: Module 1 and Module 2 provide basic convolution and pooling operations; Module 3 is used to skip connections between different modules; Module 4 and Module 5 are used to connect feature maps and, together with Module 3, construct the residual network. The combination of different modules constitutes 3 different sub-modules, which are cascaded and fused into the main network.

6. The method for detecting foreign objects in railway tunnels based on the fusion of radar point clouds and RGB images according to claim 5, characterized in that, In the fusion target detection network, Block1 and Block2 of EfficientNet-B2 are used as feature extractors to perform convolution and downsampling on the input RGB image and dense depth map respectively to obtain feature maps with consistent depth and scale. Feature maps are merged through a feature fusion layer using a 1×1 convolutional kernel. While keeping the feature map scale unchanged, cross-channel information interaction is triggered by increasing non-linearity and reducing the feature map dimension. The fused feature map is obtained by fusing the data features of the two modalities and is fed into the subsequent block of the fusion network for further extraction and downsampling of data features, finally outputting a 13×13 feature map.

7. The method for detecting foreign objects in railway tunnels based on the fusion of radar point clouds and RGB images according to any one of claims 1 to 6, characterized in that, In the fusion target detection network, a Feature Pyramid Network (FPN) structure is adopted to simultaneously learn shallow and deep features on three feature maps with 32x downsampling, 16x downsampling, and 8x downsampling to respond to and obtain multi-scale target predictions. The target location, category, and confidence score are obtained through neural network regression prediction. For each bounding box, the coordinates (x, y), width and height (w, h), and confidence score of the bounding box are predicted.

8. The method for detecting foreign objects in railway tunnels based on the fusion of radar point clouds and RGB images according to claim 7, characterized in that, The confidence level formula is expressed as follows: In the formula, This represents the intersection of the predicted bounding box and the ground truth bounding box; If the object exists in the grid, Pr(object) has a value of 1; otherwise, it has a value of 0. The loss function in the fusion target detection network consists of three parts: coordinate prediction error. coord IoU Error IoU and classification error clas ; In the formula, λ coord The weights representing coordinate prediction errors; This indicates the indicator function for the j-th object in the i-th grid. These are the prediction results for x, y, w, and h, respectively; λ nobj It is the weight of the IoU error; p(c) is the indicator function for the j-th non-object in the i-th grid; p(c) is the probability of target class c. This is the prediction result.

9. The method for detecting foreign objects in railway tunnels based on the fusion of radar point clouds and RGB images according to claim 1, characterized in that, In the Generative Adversarial Network (GAN) model, the Graph Attention Network (GAT) acquires and outputs data features, expressed by the following formula: In the Generative Adversarial Network (GAN) model, the autoencoder AE takes X as input, maps it to the latent space as variable Z, and maps the latent space variable Z to the original input space of the encoder. The error between the reconstructed X and the original input X is used as the reconstruction error. THE AE =||X-AE(X)||2; In the formula, AE(X)=D(Z), Z=E(X), and ||·||2 represents the L2 norm.

10. An electronic device, comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the railway tunnel foreign object detection method based on radar point cloud and RGB image fusion as described in any one of claims 1 to 9.

Images