A multi-sensor fusion track foreign object intrusion general perception method

Abstract

This invention discloses a multi-sensor fusion general perception method for foreign object intrusion on railway tracks, relating to the field of foreign object intrusion identification on railway tracks. The method includes building a foreign object intrusion perception model, inputting multimodal data collected by vehicle sensors into the model for processing to obtain an abnormal target perception mask, and achieving pixel-level perception and localization of abnormal targets. The foreign object intrusion perception model includes a fusion module, a feature encoding module, and a mask predictor. The fusion module is used to perform feature fusion on the input data to obtain unified features, and to model the feature distribution of the normal track area and achieve preliminary localization of the intruding foreign object. The feature encoding module is used to extract intrusion target features from the fused features to obtain pixel-level perception features. The mask predictor uses the pixel-level perception features to perform pixel-level segmentation prediction of the abnormal target and takes the union of the prediction with the preliminary localization result output by the fusion module to output the final prediction result.

Description

Technical Field

[0001] This invention relates to the field of foreign object intrusion detection in orbits, and in particular to a multi-sensor fusion general sensing method for foreign object intrusion in orbits. Background Technology

[0002] Autonomous operation of rail transit plays a crucial role in improving passenger and freight transport efficiency and ensuring line safety. However, because railway tracks are laid in an open environment, various foreign objects can potentially intrude into the tracks, seriously threatening train safety. Therefore, the primary prerequisite for achieving autonomous operation is enabling vehicles to actively detect various intruding foreign objects during operation, thereby ensuring train safety.

[0003] Existing data-driven deep learning-based closed-set target detection algorithms cannot detect unknown types of foreign objects, and perception algorithms relying on a single image sensor have poor detection capabilities when facing small targets at long distances under all weather conditions. Furthermore, due to the sporadic nature of foreign object intrusions into railway lines, it is impossible to collect a sufficient number and all types of foreign object samples. Therefore, the main problem addressed by this invention is how to achieve universal detection of various abnormal intrusion targets in rail transit scenarios, given the difficulty in collecting abnormal samples and the lack of sufficient training data.

[0004] The method proposed in "RailFDNet: A hybrid supervision and feature discrepancy enhancement model for railway anomalous object detection" employs a two-stage training approach. The first stage trains the feature extraction network using normal images to achieve initial feature extraction. The second stage generates anomalous data simulating anomalies, using this data for supervised training to further improve the model's accuracy in perceiving anomalous targets. However, this method only uses image data, thus its robustness decreases under poor lighting conditions. Furthermore, it lacks textual constraints and cannot directly output pixel-level perception results, resulting in shortcomings in the accuracy of anomalous target localization and its practical application in engineering.

[0005] While current mainstream general-purpose segmentation algorithms can segment different targets in a scene, the training datasets they use are publicly available datasets that lack railway scenes and small targets, resulting in poor segmentation performance in railway scenarios. Furthermore, these segmentation methods cannot distinguish between infrastructure and intrusive foreign objects in a railway scene, rendering their segmentation results unusable directly. As for anomaly detection algorithms, although they can locate abnormal targets in a scene after training on normal data, their output is an AUROC (Receiver Operating Characteristic under varying Uncertainty) curve, used to measure the model's overall performance under different decision thresholds, and cannot directly output a mask result for decision-making.

[0006] Furthermore, both of these methods rely heavily on single image information for analysis. Since camera imaging quality is significantly affected by the environment, fluctuations in image quality under strong light, shadow, nighttime, or inclement weather conditions can significantly impact the model's anomaly detection performance, thereby reducing the robustness and reliability of the detection.

[0007] To address these issues, a general sensing method for foreign object intrusion in orbits using multi-sensor fusion is urgently needed. Summary of the Invention

[0008] To address the aforementioned issues, this application proposes a multi-sensor fusion method for general perception of foreign object intrusion on railway tracks. The aim is to utilize multimodal data from images and point clouds, combined with simulated anomaly data, to generate corresponding text labels. During training, the constraints on the model are enhanced to make it more sensitive to anomaly target features. Finally, a mask predictor is incorporated to achieve pixel-level perception of anomaly targets while automatically ignoring normal railway infrastructure targets. This allows the algorithm to be directly applied to railway production monitoring tasks, achieving efficient, universal, and reliable perception of anomaly targets. The specific details are as follows: This application proposes a multi-sensor fusion general sensing method for foreign object intrusion in orbits, including the following steps: S1. Obtain multi-source track data of railway tracks, and preprocess the multi-source track data to obtain input data. The multi-source track data includes track point cloud data and track image data. S2. Establish a foreign object intrusion detection model for the track; The orbital foreign object intrusion detection model includes a fusion module, a feature encoding module, and a mask predictor. The fusion module is used to perform feature fusion on the input data to obtain unified features, and to model the feature distribution of the normal orbit area and achieve preliminary location of the intruding foreign object; The feature encoding module is used to extract intrusion target features from the unified and fused features to obtain pixel-level perceptual features. ; The mask predictor is used to utilize pixel-level perceptual features. Perform pixel-level segmentation prediction of abnormal targets and combine it with the preliminary localization results output by the fusion module to output the final prediction result; S3. Input the input data into the orbital foreign object intrusion perception model for processing to obtain the abnormal target perception mask, thereby achieving pixel-level perception and localization of abnormal targets.

[0009] Preferably, the preprocessing of multi-source data to obtain input data specifically includes dimensional transformation of 3D orbital point cloud data to obtain 2D array data, specifically including: Let the data of the two-dimensional array be denoted as ; Based on the projection formula, points in the orbital point cloud data are projected onto a two-dimensional plane, and the points are obtained in... Corresponding coordinates The expression for the projection formula is: ; In the formula, i This represents the data index of points in the 3D point cloud acquired by the lidar, [ ] represents the three-dimensional coordinates of the point cloud. This represents the corresponding 3D point cloud in a 2D array data. The corresponding data storage location, K This refers to the camera's intrinsic parameters, specifically: ; Let be the equivalent focal length in the x-direction. Let be the equivalent focal length in the y-direction. Let x be the x-coordinate of the intersection of the optical axis and the image plane on the image. The vertical coordinate of the intersection of the optical axis and the image plane on the image. Each position The filling is a three-channel array resulting from the normalization of point cloud coordinates. The expression for the three-channel array is: ; in, , , These are the three-dimensional spatial coordinates of the i-th point in the dot cloud, respectively. , , These are the largest x-axis, y-axis, and z-axis coordinates of all points in the point cloud, respectively. , , , , and z are the smallest x-axis, y-axis, and z-coordinates of all points in the point cloud, respectively.

[0010] Preferably, the specific content of the fusion module includes: The input data is subjected to feature extraction and weighted fusion to obtain multimodal features; A unified feature representation is obtained by probabilistic distribution modeling based on multimodal features; The input data includes two-dimensional point cloud processing data and track image data; During the training process, the fusion module completes the probability distribution modeling of the abnormal scenario based on the input data without abnormal data to obtain the standard distribution; In application, the unified feature represents the feature distribution output by the fusion module after the point cloud and image data collected by the vehicle-mounted sensor are sent into it. If the data contains anomalous targets, the output feature distribution will differ from the standard distribution. The difference between the output feature distribution and the standard distribution can be used to achieve preliminary localization (when data containing anomalous targets is sent in, the anomalous score of the output result will be high, and the result can be obtained directly).

[0011] Preferably, the specific content of extracting features from the input data and weighting and fusing them to obtain multimodal features includes: Camera images and point cloud images Initial feature extraction of unimodal data is performed using WideResNet, followed by unimodal feature modeling of image and point cloud data using normalized flow to obtain image features. and point cloud features ; By fusion weights To balance information density, feature weighting fusion is performed by controlling the proportion of different weights in the fused features. The expression for feature weighting fusion is as follows: .

[0012] Then, the normalized flow pairs are used to fuse features to further model the feature distribution of the normal track region.

[0013] Preferably, the feature encoding module is configured with a text encoder and a feature fusion encoder, specifically including: Using anomaly-free orbital multi-source data, simulate the generation of orbital multi-source data containing anomalous targets, along with corresponding mask labels; Multi-source orbital data containing anomalous targets includes point cloud data containing anomalous targets and image data containing anomalous targets. Text description labels are generated by utilizing the relationship between the location of abnormal targets in abnormal data and railway tracks, as well as their proximity to distance sensors. By utilizing multi-source orbital data containing anomalous targets and using textual description labels as constraints, the feature encoding module can be trained. Specifically, the text encoder inputs text description labels, while the feature fusion encoder inputs fused features extracted by the frozen feature fusion module. The feature encoding module outputs pixel-level perceptual features. .

[0014] Preferably, a fusion feature encoder is used to process multi-source track data to obtain pixel-level sensing features. The specific content includes: Use convolution to process the input features The merged and flattened features are added to the positional encoding and then fed into the transformer module to obtain global features. ; The position encoding uses multi-frequency sine and cosine basis functions for generation; The fusion feature encoder includes: and After normalization, the residues are added together using the concept of residual joins to obtain the result. This is used for subsequent pixel-level perception result prediction; Will After pooling, features are merged and compressed to obtain multimodal global representative features. .

[0015] Preferably, this includes: during the training process of the fusion feature encoder, text features are obtained by encoding text description labels based on the text encoder. Cosine similarity is used for loss calculation, which enables the fusion feature encoder to accurately extract the features of abnormal targets. The expression for loss calculation is: ; in, It indicates a loss.

[0016] Preferably, the input to the multi-branch fusion mask predictor includes multimodal data fusion features. and preliminary location results; for Feature extraction is performed by progressive upsampling and convolution channel compression. Then, a probability matrix is ​​obtained by convolution with 2 output channels. The mask attribute is determined by comparing the probability of the two channels at the same position: 0 represents a normal target and 1 represents an abnormal target. The preliminary location result refers to the preliminary location mask; The multi-branch fusion mask predictor fuses the mask obtained by probability comparison with the preliminary positioning mask and takes the union to obtain the final prediction result.

[0017] Preferably, the parameters of the feature fusion module are frozen during the training of the fusion feature encoder.

[0018] Preferably, during the training of the mask predictor, the parameters of the other modules are frozen to achieve supervised training of the predictor.

[0019] In actual use, all parameters will be frozen, and the images and point cloud data collected by the vehicle sensors will be input into the model to achieve pixel-level perception and prediction of intrusive foreign objects.

[0020] In summary, the multi-sensor fusion general sensing method for foreign object intrusion in orbit, as proposed in this invention, has the following advantages compared to traditional technologies: 1. By using camera intrinsic parameters and size normalization, irregular 3D radar point cloud data is converted into regular three-channel 2D data. While ensuring that the original 3D coordinate information is not lost, the point cloud data is transformed into a data form that can be directly used in the perception model. This can reduce the difficulty of data preprocessing and realize point-to-point matching between multimodal data. 2. A multi-level feature fusion method is used to perform preliminary feature extraction on single-modal data using different streaming modules to obtain the distribution of non-abnormal features in different modal data, and then further feature extraction is performed on the fused features using streaming modules to achieve effective modeling and extraction of features for normal scenes. 3. A method for generating region description text based on the relationship between the simulated anomaly target location and the rail and the distance to the sensor, through simulated anomaly generation and region division; 4. Using the generated region description text as labels to supervise the model enables the algorithm to more accurately locate abnormal information in multimodal features; 5. Treating line foreign object intrusion detection as a general abnormal target perception problem, pixel-level segmentation perception of abnormal targets is achieved through multimodal data feature-level fusion combined with text constraint training.

[0021] The technical method of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0022] Figure 1 This is the overall structure of the multi-sensor fusion general sensing system for foreign object intrusion in orbit according to the present invention; Figure 2 This is a schematic diagram illustrating the textual description of the present invention. Figure 2 In the diagram, (a) represents the point cloud data input to the model. Figure 2 (b) in the figure represents the image data input to the model. Figure 2 (c) in the text represents the keywords used for generating the constraint text; Figure 3 This is a schematic diagram of the fusion feature encoder of the present invention; Figure 4 This is a schematic diagram of the predictor of the present invention; Figure 5 This is a comparison chart of the results of the embodiments of the present invention. Detailed Implementation

[0023] The technical method of the present invention will be further described below with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of this application.

[0024] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the scope of this application and its application or use.

[0025] Techniques, systems, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, they should be considered part of the instruction manual.

[0026] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0027] Unless otherwise defined, the technical or scientific terms used in this invention shall have the ordinary meaning as understood by one of ordinary skill in the art to which this invention pertains.

[0028] This invention provides a multi-sensor fusion method for general perception of foreign object intrusion on railway tracks. In the face of the diversity and sporadic occurrence of foreign objects intruding in railway scenarios, the main purpose of this invention is to identify targets that do not belong to the normal scenario, that is, to regard all types of abnormal intrusion objects as the same type of target.

[0029] This invention comprises three parts: a feature fusion module, a feature encoding module, and a mask predictor. The perception method training is divided into three stages: The first stage uses anomaly-free point cloud and image data to train the feature fusion module based on normalized flow, completing the feature distribution modeling for normal track scenes. When the data contains anomaly targets, it can achieve preliminary localization and feature extraction of the anomaly targets. The second stage first simulates the generation of point cloud and image data containing anomaly targets, along with corresponding mask labels, using the anomaly target's generation location relative to the rail and its distance from the sensor to generate text description labels. Then, the feature fusion module from the first stage is frozen, and the features extracted from the simulated anomaly point cloud and image data are fed into the feature encoding module. Combined with text constraints, the feature encoding module is trained to achieve further refined feature extraction of anomaly targets. The third stage freezes the fusion module and the feature encoding module, and the features output from the feature encoding module are fed into the mask predictor. The predictor is trained under the constraints of the mask labels to achieve pixel-level perception of anomaly targets. During testing, all modules are frozen to capture images and point cloud data from the input sensors. The method can output pixel-level segmentation perception results for abnormal targets.

[0030] Example 1 The overall structure of the technical solution is as follows Figure 1 As shown, the model mainly consists of three parts: a fusion module, a feature encoding module, and a mask predictor, and corresponds to three training stages, as detailed below: The fusion module's main function is to perform feature-level fusion of image data and point cloud data. It consists of a feature extraction backbone network based on the first three layers of WideResNet and a flow module based on normalized flow. Since the data generated by LiDAR is variable-length 3D point cloud data, which cannot be directly used by the model, this invention designs a point cloud data conversion method. This method can transform variable-length point clouds into regular 2D data without losing 3D information. The specific process is as follows: Let the data of the two-dimensional array be denoted as ; Based on the projection formula, points in the orbital point cloud data are projected onto a two-dimensional plane, and the points are obtained in... Corresponding coordinates The expression for the projection formula is: ; In the formula, i This represents the data index of points in the 3D point cloud acquired by the lidar, [ ] represents the three-dimensional coordinates of the point cloud. This represents the corresponding 3D point cloud in a 2D array data. The corresponding data storage location, KThis refers to the camera's intrinsic parameters, specifically: ; Let be the equivalent focal length in the x-direction. Let be the equivalent focal length in the y-direction. Let x be the x-coordinate of the intersection of the optical axis and the image plane on the image. The vertical coordinate of the intersection of the optical axis and the image plane on the image. Each position The filling is a three-channel array resulting from the normalization of point cloud coordinates. The expression for the three-channel array is: ; in, , , These are the three-dimensional spatial coordinates of the i-th point in the dot cloud, respectively. , , These are the largest x-axis, y-axis, and z-axis coordinates of all points in the point cloud, respectively. , , , , and z are the smallest x-axis, y-axis, and z-coordinates of all points in the point cloud, respectively.

[0031] Common point cloud transformation methods calculate the distance of a point from the sensor using the three-dimensional coordinates of the point cloud and fill it into a two-dimensional array as the result. In contrast, the method in this application retains the three-dimensional coordinates of the point cloud while realizing the data dimension transformation, without causing information loss, and can provide more sufficient information for perception algorithms.

[0032] Within the fusion module, a multi-level feature fusion mechanism is employed, such as... Figure 1 As shown, camera image and point cloud images After initial feature extraction using WideResNet, the data are fed into streaming modules 1, 2, 3 and 4, 5, 6 respectively for single-modal feature modeling of image and point cloud data, thereby obtaining image features. and point cloud features This achieves preliminary feature extraction. To fully utilize features from different modalities and further achieve more accurate modeling of feature distributions in anomaly-free scenes, this invention employs feature fusion via feature addition, using fusion weights... The proportions of different weights in the fused features are controlled to balance information density. The specific process of feature weighted fusion is shown in the following equation: .

[0033] The multimodal features after addition and fusion are input into the input stream modules 7, 8, and 9 to model the distribution of non-abnormal multimodal features, so as to generate a unified feature representation for subsequent result prediction.

[0034] The fusion module uses anomaly-free multimodal data during training, so it uses maximum likelihood estimation to calculate the loss function and achieve self-supervised training.

[0035] The fusion module can model the feature distribution of normal scenes based on data without anomalies. When data containing anomalous intrusion targets is fed into the module, it can generate a significant response to the location of the anomalous targets.

[0036] Text description labels are generated by utilizing the relationship between the location of abnormal targets in abnormal data and railway tracks, as well as their proximity to distance sensors.

[0037] To further improve the model's accuracy in locating intruding foreign objects, this invention simulates and generates, as follows: Figure 2 The simulated point cloud and image containing the foreign object are shown on the left side and in the middle. Simultaneously, to divide the track area longitudinally into M regions according to the distance of the foreign object from the sensor, M is 5 in this invention, and the distance description is as follows: Figure 2 As shown in the upper right corner, the relationship with the left and right rails is divided into N regions horizontally. In this invention, N is 3, and the relationship description is as follows: Figure 2 As shown in the bottom right corner. After the location and relationship descriptors are generated, corresponding text labels are generated using the region text label template, such as "Detected anomaly close to the right track in far away" and "No foreign objects on the track," to describe foreign objects in the scene.

[0038] To extract intrusion target features from fused features, this application designs the following... Figure 3 The encoder shown first uses convolution to merge the input features, then flattens them and adds them to the positional encoding before feeding them into the transformer module to obtain global features. The position encoding uses multi-frequency sine and cosine basis functions to generate the position, which maps the absolute position to linearly modelable relative position information, enabling the Transformer to have the ability to sense order and model long distances without recursion or convolution.

[0039] The fusion feature encoder has two main functions: first, it combines... and After normalization, the residues are added together using the concept of residual joins to obtain the result. This is used for subsequent pixel-level perception result prediction. The second is to... After pooling, feature merging is performed, and a learnable projector is designed to compress features to obtain multimodal globally representative features. .

[0040] During encoder training, the parameters of the fusion module are frozen. Region description text labels are fed into the text encoder to improve the model's accuracy in locating abnormal targets. During prediction, the text branch does not participate in the calculation. The loss calculation formula is as follows: .

[0041] S6. Construct a multi-branch fusion mask predictor utilizing pixel-level perceptual features. Based on the preliminary localization results, anomaly target perception mask is predicted, enabling pixel-level perception and localization of anomaly targets.

[0042] like Figure 4 As shown, the input to the mask predictor consists of two parts: the first part is the multimodal data fusion feature. ,for Feature extraction is performed by progressive upsampling and channel compression using convolution. Then, a probability matrix is ​​obtained by convolution with 2 output channels. By comparing the probability of the two channels at the same position, channel 0 and channel 1 represent the probability of the pixel being normal or abnormal, respectively. If the probability of channel 0 at the position is greater than the probability of channel 1, the prediction result at that position is 0, otherwise it is 1. 0 represents a normal target and 1 represents an abnormal target.

[0043] The other branch input is the preliminary localization mask obtained from the fusion module's impact on abnormal targets. In the predictor, the mask obtained through probability comparison is fused with the preliminary localization mask and the union is taken to obtain the final prediction result.

[0044] The initial localization mask is extracted by setting an anomaly threshold. If the probability of a pixel being an abnormal target is greater than the anomaly threshold, it is judged as 1, otherwise it is 0. The initial localization mask and the predictor output mask are both 0, and the final result is 0. Otherwise, it is 1.

[0045] During the training of the mask predictor, the parameters of the other modules are frozen. In order to achieve supervised training of the predictor, this invention uses the mask labels obtained during the simulation data generation process to supervise the training of the predictor.

[0046] Example 2 Table 1 Ablation Experiment

[0047] The ablation experiment results are shown in Table 1. The results clearly show that adding radar data to the model significantly improves the localization accuracy of anomalous targets, indicating that 3D spatial information can effectively assist in anomaly detection. Furthermore, when pre-training with text containing region descriptions to guide the fusion feature encoder, the model's performance in anomaly target localization is significantly improved, fully demonstrating the effectiveness of region-based text supervision constraints in guiding the model to focus on anomalous regions and improving localization accuracy. These results indicate that fusing multimodal information and text region-guided supervision can synergistically enhance the model's anomaly detection and localization capabilities in railway scenarios.

[0048] This invention compares itself with two mainstream open-world target perception methods, SAM and YOLO E, from... Figure 5 As can be seen, although SAM and YOLO E use a large amount of data for pre-training and have general target perception capabilities in common scenarios, their detection performance is poor due to the small size and diverse shapes of foreign objects in rail transit scenarios. In contrast, the method in this application can achieve better localization of abnormal targets in both rainy and sunny weather, and will not produce false alarms in scenarios without abnormalities.

[0049] In summary, compared to existing technologies, this invention firstly enhances the robustness of the algorithm by fusing multi-sensor data, combining point cloud and image data, and also improves the detection accuracy of anomalous targets by leveraging complementary multi-source information. Secondly, through simulated data generation technology, the location and size of foreign objects can be controlled, and corresponding text labels can be automatically generated, allowing for better constraints on the model during training. Finally, by designing a mask predictor, the model can directly provide pixel-level perception results for anomalous targets, improving efficiency and safety in rail transit production and operation. In conclusion, this invention significantly enhances the feasibility and practical value of its applications while improving detection performance.

[0050] Finally, it should be noted that the above embodiments are only used to illustrate the technical methods of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical methods of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical methods to deviate from the spirit and scope of the technical methods of the present invention.

Claims

1. A multi-sensor fusion method for general sensing of foreign object intrusion in orbits, characterized in that, Includes the following steps: S1. Obtain multi-source track data of railway tracks, and preprocess the multi-source track data to obtain input data. The multi-source track data includes track point cloud data and track image data. S2. Establish a foreign object intrusion detection model for the track; The orbital foreign object intrusion detection model includes a fusion module, a feature encoding module, and a mask predictor. The fusion module is used to perform feature fusion on the input data to obtain unified features, and to model the feature distribution of the normal orbit area and achieve preliminary location of the intruding foreign object; The feature encoding module is used to extract intrusion target features from the fused features to obtain pixel-level perceptual features. ; The mask predictor is used to utilize pixel-level perceptual features. Perform pixel-level segmentation prediction of abnormal targets and take the union with the preliminary localization results output by the fusion module to output the final prediction result; S3. Input the multimodal data collected by the vehicle sensors into the track foreign object intrusion perception model for processing to obtain the abnormal target perception mask, thereby achieving pixel-level perception and localization of abnormal targets.

2. The multi-sensor fusion general sensing method for foreign object intrusion in orbits according to claim 1, characterized in that, The specific content of the input data obtained by preprocessing multi-source data includes dimensional transformation of 3D orbital point cloud data to obtain 2D array data, specifically including: Let the data of the two-dimensional array be denoted as ; Based on the projection formula, points in the orbital point cloud data are projected onto a two-dimensional plane, and the points are obtained in... Corresponding coordinates The expression for the projection formula is: ； In the formula, i This represents the data index of the points in the 3D point cloud acquired by the lidar, [ ] represents the three-dimensional coordinates of the point cloud. This represents the corresponding 3D point cloud in a 2D array data. The corresponding data storage location, K This refers to the camera's intrinsic parameters, specifically: ； Let be the equivalent focal length in the x-direction. Let be the equivalent focal length in the y-direction. Let x be the x-coordinate of the intersection of the optical axis and the image plane on the image. The vertical coordinate of the intersection of the optical axis and the image plane on the image. Each position The filling is a three-channel array resulting from the normalization of point cloud coordinates. The expression for the three-channel array is: ； in, , , These are the three-dimensional spatial coordinates of the i-th point in the dot cloud. , , These are the largest x-axis, y-axis, and z-axis coordinates of all points in the point cloud. , , , , and z are the smallest x-axis, y-axis, and z-coordinates of all points in the point cloud, respectively.

3. The multi-sensor fusion general sensing method for foreign object intrusion in orbits according to claim 1, characterized in that, The specific contents of the fusion module include: The input data is subjected to feature extraction and weighted fusion to obtain multimodal features; A unified feature representation is obtained by probabilistic distribution modeling based on multimodal features; The input data includes two-dimensional point cloud processing data and track image data; During the training process, the fusion module completes the probability distribution modeling of the abnormal scenario based on the input data without abnormal data to obtain the standard distribution; In application, the unified feature represents the feature distribution output by the fusion module after the point cloud and image data collected by the vehicle-mounted sensor are sent into it. If the data contains abnormal targets, the output feature distribution will differ from the standard distribution. The initial location can be achieved based on the difference between the output feature distribution and the standard distribution.

4. The multi-sensor fusion general sensing method for foreign object intrusion in orbits according to claim 1, characterized in that, The specific content of extracting features from the input data and weighting and fusing them to obtain multimodal features includes: Camera images and point cloud images Initial feature extraction of unimodal data is performed using WideResNet, followed by unimodal feature modeling of image and point cloud data using normalized flow to obtain image features. and point cloud features ; By fusion weights To balance information density, feature weighting fusion is performed by controlling the proportion of different weights in the fused features. The expression for feature weighting fusion is as follows: ； Then, the normalized flow pairs are used to fuse features to further model the feature distribution of the normal track region.

5. The multi-sensor fusion general sensing method for foreign object intrusion in orbits according to claim 1, characterized in that, The feature encoding module is configured with a text encoder and a feature fusion encoder, specifically including: Using anomaly-free orbital multi-source data, simulate the generation of orbital multi-source data containing anomalous targets, along with corresponding mask labels; Multi-source orbital data containing anomalous targets includes point cloud data containing anomalous targets and image data containing anomalous targets. Text description labels are generated by utilizing the relationship between the location of abnormal targets in abnormal data and railway tracks, as well as their proximity to distance sensors. By utilizing multi-source orbital data containing anomalous targets and using textual description labels as constraints, the feature encoding module can be trained. Specifically, the text encoder inputs text description labels, while the feature fusion encoder inputs fused features extracted by the frozen feature fusion module. The feature encoding module outputs pixel-level perceptual features. .

6. The multi-sensor fusion general sensing method for foreign object intrusion in orbits according to claim 1, characterized in that, A fusion feature encoder is used to process multi-source track data to obtain pixel-level sensing features. The specific content includes: Use convolution to process the input features The merged and flattened features are added to the positional encoding and then fed into the transformer module to obtain global features. ; The position coding is generated using multi-frequency sine and cosine basis functions; The fusion feature encoder includes: and After normalization, the residuals are added and fused to obtain the result. This is used for subsequent pixel-level perception result prediction; Will After pooling, features are merged and compressed to obtain multimodal global representative features. .

7. The multi-sensor fusion general sensing method for foreign object intrusion in orbits according to claim 1, characterized in that, include: During the training of the fusion feature encoder, text features are obtained by encoding text description labels based on the text encoder. Cosine similarity is used for loss calculation, which enables the fusion feature encoder to accurately extract the features of abnormal targets. The expression for loss calculation is: ； in, It indicates a loss.

8. The multi-sensor fusion general sensing method for foreign object intrusion in orbits according to claim 1, characterized in that, The input to the multi-branch fusion mask predictor includes multimodal data fusion features. Preliminary positioning results of the fusion module; for Feature extraction is performed by progressive upsampling and convolution channel compression. Then, a probability matrix is ​​obtained by convolution with 2 output channels. The mask attribute is determined by comparing the probability of two channels at the same position: 0 represents a normal target and 1 represents an abnormal target. The preliminary location result refers to the preliminary location mask; The multi-branch fusion mask predictor fuses the mask obtained by probability comparison with the preliminary positioning mask and takes the union to obtain the final prediction result.

9. A multi-sensor fusion method for general sensing of foreign object intrusion in orbits according to claim 7, characterized in that, During the training of the fusion feature encoder, the parameters of the feature fusion module are frozen.

10. A multi-sensor fusion general sensing method for foreign object intrusion in orbits according to claim 9, characterized in that, During the training of the mask predictor, the parameters of the other modules are frozen, thus enabling supervised training of the predictor. In actual use, all parameters will be frozen, and the images and point cloud data collected by the vehicle sensors will be input into the model to achieve pixel-level perception and prediction of intrusive foreign objects.

Images