A foreign matter detection method, system, device and storage medium in a railway scene

Abstract

The application provides a foreign matter detection method, system and device in a railway scene and a storage medium, and belongs to the technical field of computer vision and railway safety monitoring. The method comprises the following steps: synchronously collecting visible light, low-light and infrared images in front of a train, determining corresponding track regions of interest on various images through geometric projection; extracting color texture, edge contour and temperature distribution features in respective regions, performing brightness fitness evaluation on the visible light image based on a double Sigmoid function; performing signal-to-noise ratio evaluation on the low-light image based on gradient information entropy and noise intensity; performing thermal contrast evaluation on the infrared image based on temperature distribution standard deviation; dynamically calculating fusion weights according to the confidence of the three, adaptively weighting and fusing multi-modal features to generate fusion features. The features are input into a pre-trained classifier to complete accurate detection and classification of track foreign matters. The system accuracy and environmental adaptability are significantly improved, and real all-weather safety monitoring is realized.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and railway safety monitoring technology, specifically relating to a method, system, equipment and storage medium for foreign object detection in railway scenarios. Background Technology

[0002] With the development of railway transportation towards high speed and heavy load, train operation safety faces severe challenges. Obstructions ahead of the track (such as falling rocks, fallen trees, and intrusion by people or animals) are one of the main hidden dangers leading to major accidents. Therefore, deploying a forward-looking sensing system at the front of the train to achieve all-weather, real-time track monitoring and early warning has become crucial to ensuring operational safety.

[0003] To address this challenge, existing technologies utilize machine learning techniques (CNN, Random Forest) to fuse and classify multi-source image information. Visible light imaging, in daylight and under good lighting conditions, provides high-resolution images rich in texture and detail, facilitating the identification and classification of foreign objects. Infrared thermal imaging works based on the thermal radiation of objects, independent of ambient light, and can effectively detect organisms or heat-generating devices with temperature differences at night, in tunnels, and in adverse weather conditions. Low-light imaging aims to fill the blind spots of visible light in low illumination by enhancing weak ambient light, providing contour and scene information at night.

[0004] However, in the complex all-weather operation environment of railways, visible light can only be used for a binary judgment of usability or unusability based on rough illumination conditions. In scenes with gradually changing illumination, such as dawn, dusk, and tunnel entrances, it is impossible to provide a smooth and continuous reliability score for image quality, leading to abrupt changes in fusion decisions near the threshold and unstable model output. Low-light images are assumed to be reliable under low illumination, lacking online evaluation of their actual effective information content (such as sharpness and noise resistance). When image quality is severely degraded due to motion blur, strong glare, or excessive noise, the system still treats it as valid information for fusion, introducing the risk of misjudgment. For infrared images, it is impossible to scientifically quantify the salience of target thermal features in scenes with complex fluctuations in background temperature, resulting in weak detection capabilities for low-contrast thermal targets or targets in complex thermal backgrounds. Summary of the Invention

[0005] To address the aforementioned problems, this invention provides a method, system, device, and storage medium for foreign object detection in railway scenarios.

[0006] To achieve the above objectives, the present invention provides a foreign object detection method in a railway scenario, comprising: Simultaneously acquire visible light images, low-light images, and infrared thermal radiation images in front of the train, and record the train's position and posture information at the time of acquisition.

[0007] Based on the train's pose information and the pre-stored electronic track map, the target physical track section in front of the train at the current acquisition time is determined, and the target physical track section is projected onto the visible light image, low light image, and infrared thermal radiation image respectively to obtain the visible light ROI, low light ROI, and infrared ROI; the color texture features of the visible light ROI region are extracted, the edge contour features of the low light ROI region are extracted, and the temperature distribution features of the infrared ROI region are extracted.

[0008] Calculate the average brightness value of the visible light ROI region, and determine the confidence level of the visible light ROI region based on the average brightness value; process the low-light image into a gradient magnitude map, and statistically analyze the information entropy and local noise intensity of the gradient magnitude map, and determine the confidence level of the low-light ROI based on the information entropy and local noise intensity; calculate the global standard deviation of the temperature values ​​of all pixels in the infrared ROI region, and determine the confidence level of the infrared ROI based on the global standard deviation.

[0009] The weights of color texture features, edge contour features, and temperature distribution features are determined by the confidence levels of the visible light ROI, low light ROI, and infrared ROI, respectively. The color texture features, edge contour features, and temperature distribution features are fused according to the weights to obtain weighted fused features. The weighted fused features are then input into a pre-trained classifier to complete the detection of foreign objects on railway tracks.

[0010] Preferably, the average brightness value of the visible light ROI region is calculated. Based on the average brightness value, a confidence level of the average brightness value of the visible light ROI region is calculated using a double sigmoid function with low light threshold and overexposure threshold as key parameters. The double sigmoid function is configured to output a confidence level value close to 1 when the brightness is between the low light threshold and the overexposure threshold, and to output a confidence level value close to 0 when the brightness is lower than the low light threshold or higher than the overexposure threshold. The obtained confidence level is the confidence level of the visible light ROI region.

[0011] Preferably, the low-light image is processed into a gradient magnitude map using the Sobel gradient operator, and the local variance of the flat region of the low-light image is calculated to obtain the noise intensity; the information entropy of the gradient magnitude map is calculated, and the information entropy is corrected using the noise intensity to obtain the texture entropy; the texture entropy is compared with a preset sharpness reference entropy value, and the confidence level of the low-light ROI is determined according to the ratio of texture entropy to reference entropy value; the confidence level increases with the increase of texture entropy, and the confidence level reaches a saturation maximum value when the texture entropy reaches or exceeds the reference entropy value.

[0012] Preferably, the global standard deviation of the temperature values ​​of all pixels within the infrared ROI region is calculated, and the confidence level of the infrared ROI is determined based on the global standard deviation. Specifically, this includes: calculating the standard deviation of the temperature values ​​of all pixels within the infrared ROI region, establishing a monotonically increasing mapping function based on the standard deviation, and calculating the confidence level of the infrared ROI; the mapping function is configured such that: when the standard deviation is larger, a higher confidence level is output; when the standard deviation approaches zero, the lowest confidence level is output.

[0013] Preferably, before simultaneously acquiring visible light images, low-light images, and infrared thermal radiation images in front of the train, the method further includes: visual calibration of the devices acquiring the three types of images. Specifically, based on the train's pose information and a pre-stored electronic track map, the angular deviation between the extension direction of the track in front of the train and the train's current heading is obtained; according to the angular deviation, the pan-tilt unit carrying the devices for acquiring the visible light images, low-light images, and infrared thermal radiation images is controlled to rotate, so that the visual axis of the acquisition devices is aligned with the track in front, thereby completing the visual calibration of the devices for acquiring the three types of images.

[0014] Preferably, calculating the angle deviation specifically includes: obtaining the coordinates of a track point at a predetermined distance ahead from the electronic track map based on the current position of the train, and calculating the azimuth angle of the track point relative to the current position of the train; the angle deviation is the difference between the azimuth angle and the train's heading angle.

[0015] Preferably, the line-of-sight alignment specifically includes: based on the electronic track map, presetting the gimbal angle in key areas of the track curve, and establishing a track node-preset position mapping table; during train operation, according to the real-time position of the train, calling the corresponding preset position coordinates from the track node-preset position mapping table, and driving the gimbal to jump quickly; when the gimbal points to the preset position, calculating continuous angle corrections based on the real-time relative position of the train and the track, and driving the gimbal to perform sub-second dynamic tracking to achieve line-of-sight alignment.

[0016] This invention also provides a foreign object detection system for railway scenarios, comprising: The data acquisition module is used to simultaneously acquire visible light images, low-light images, and infrared thermal radiation images in front of the train, and record the train's position and posture information at the time of acquisition.

[0017] The calculation module is used to determine the target physical track segment ahead of the train at the current acquisition time based on the train pose information and the pre-stored electronic track map, and to project the target physical track segment onto the visible light image, low-light image, and infrared thermal radiation image respectively to obtain the visible light ROI, low-light ROI, and infrared ROI; extract the color texture features of the visible light ROI region, extract the edge contour features of the low-light ROI region, and extract the temperature distribution features of the infrared ROI region; calculate the average brightness value of the visible light ROI region, and determine the confidence level of the visible light ROI region based on the average brightness value; process the low-light image into a gradient magnitude map, and statistically analyze the information entropy and local noise intensity of the gradient magnitude map, and determine the confidence level of the low-light ROI based on the information entropy and local noise intensity; calculate the global standard deviation of the temperature values ​​of all pixels in the infrared ROI region, and determine the confidence level of the infrared ROI based on the global standard deviation.

[0018] The detection module is used to determine the weights of color texture features, edge contour features, and temperature distribution features based on the confidence levels of the visible light ROI, low light ROI, and infrared ROI, respectively; to fuse the color texture features, edge contour features, and temperature distribution features according to the weights to obtain weighted fused features; and to input the weighted fused features into a pre-trained classifier to complete the detection of foreign objects on the railway track.

[0019] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement any of the steps in the foreign object detection method in the railway scenario.

[0020] The present invention also provides a computer-readable storage medium storing a computer program, which, when loaded by a processor, is capable of executing any of the steps in the foreign object detection method in the railway scenario.

[0021] This invention provides a foreign object detection method for railway scenarios with the following beneficial effects: By calculating the information entropy of the image gradient map and introducing a signal-to-noise ratio (SNR) evaluation mechanism, this invention directly quantifies the effective texture clarity and structural richness of low-light images. This method can accurately remove random noise interference introduced by high gain, thereby precisely identifying image quality degradation caused by motion blur, strong light glare, or excessively low SNR. Even when the image brightness is sufficient but filled with noise or blurred content, it can correctly output low confidence, thus preventing low-quality data from contaminating fusion decisions. Simultaneously, by introducing a global standard deviation to obtain a temperature statistical significance index, the evaluation results can reflect the distinguishability of the target against a specific background, significantly improving the accuracy of reliability assessment for low-contrast thermal targets or targets in complex thermal backgrounds. Furthermore, the reliability of visible light, low-light, and infrared sensors across different physical dimensions (illuminance adaptability, structural clarity, and thermal contrast significance) is uniformly mapped to a comparable confidence scale, fundamentally eliminating the inherent environmental blind spots in key challenging operating conditions and achieving high robustness and high accuracy in foreign object detection. Attached Figure Description

[0022] To more clearly illustrate the embodiments and design schemes of the present invention, the accompanying drawings required for this embodiment will be briefly described below. The drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart of a foreign object detection method in a railway scenario according to an embodiment of the present invention; Figure 2 This is a diagram of a train inspection system according to an embodiment of the present invention; Figure 3 This is a schematic diagram illustrating the principle of real-time angle correction in an embodiment of the present invention. Figure 4 This is a daytime scene captured by a low-light camera according to an embodiment of the present invention; Figure 5 A daytime scene captured by an infrared camera according to an embodiment of the present invention; Figure 6 A daytime scene captured by a visible light camera according to an embodiment of the present invention; Figure 7 The tunnel scene captured by the low-light camera in an embodiment of the present invention; Figure 8 The image shows a scene of entering a tunnel captured by an infrared camera according to an embodiment of the present invention. Figure 9 The tunnel scene captured by a visible light camera according to an embodiment of the present invention; Figure 10The tunnel exit scene captured by the low-light camera in an embodiment of the present invention; Figure 11 The image shows an exiting tunnel scene captured by an infrared camera according to an embodiment of the present invention. Figure 12 The tunnel exit scene captured by a visible light camera according to an embodiment of the present invention; Figure 13 Night scene captured by a low-light camera according to an embodiment of the present invention; Figure 14 A night scene captured by an infrared camera according to an embodiment of the present invention; Figure 15 Night scene captured by a visible light camera according to an embodiment of the present invention. Detailed Implementation

[0024] To enable those skilled in the art to better understand and implement the technical solutions of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and should not be construed as limiting the scope of protection of the present invention.

[0025] This invention provides a foreign object detection method in a railway scenario, specifically as follows: Figure 1 As shown, it includes the following steps: S1. Simultaneously acquire visible light images, low-light images, and infrared thermal radiation images in front of the train, and record the train's position and posture information at the time of acquisition.

[0026] like Figure 2 As shown, a visible light camera, a low-light camera, and an infrared thermal imager are rigidly connected at the front of the train. These three devices are controlled by a single hardware trigger signal to simultaneously expose and capture images of the scene ahead at the same moment. Simultaneously, the train's pose information at the moment of exposure, including latitude and longitude, heading angle, and pitch angle, is read and recorded from the train's integrated navigation system (GNSS / IMU).

[0027] The train inspection system integrates an infrared thermal imaging system, a high-definition imaging and encoding system, and a pan-tilt control system. Through the high-definition imaging and encoding module, it can provide megapixel-level video and achieve PTZ control, including 360° horizontal rotation of the pan-tilt unit, enabling true day and night all-around video surveillance. Utilizing thermal imaging technology, it is not limited by light sources and produces clear images; it features a day / night low-light color-to-black camera; a omnidirectional variable-speed pan-tilt unit, achieving all-around monitoring without blind spots and high-precision positioning; it is simple to operate, easy to use and maintain; passive thermal imaging provides good concealment; its professionally designed casing is aesthetically pleasing, lightweight, robust, high-temperature resistant, corrosion-resistant, waterproof, and acid rain resistant; the entire unit has low power consumption, generates little heat, and has stable performance. The new time synchronization system uses NTP time synchronization, and for certain brand cameras, it still uses the brand's SDK to pull the stream and read hardware and software timestamp information. However, for low-light cameras, it uses information embedded in the SEL of the video stream to read the OSD timestamp and the hardware timestamp within the RTP packet.

[0028] The low-light camera streaming latency trade-off is addressed by using FFmpeg's avformat_open_input technique, which employs two streaming functions. While TCP is relatively reliable, it introduces approximately a one-second delay for low-light cameras, whereas UDP offers lower latency and faster speed.

[0029] Synchronization was required due to differing start timestamps and slight clock drift in a certain brand of camera. A unified time synchronization was needed, using the host's system time as the reference for NTP synchronization, with time calibration every 5 minutes. The final code project implemented time synchronization for three devices, video stream retrieval and video recording capture (every 5 minutes), and the final reading and storage of timestamp data (hardware and software timestamps).

[0030] Based on the analysis of the control requirements of the train vision system and combined with basic parameters, the key performance requirements of the gimbal and the verification conclusions are as follows: Control accuracy: Image offset trigger threshold 0.5° (corresponding to track projection 8.73m), minimum control accuracy ±0.1° (corresponding to track projection 1.75m). The turntable supports ±0.1° step control, fully meeting the accuracy requirements.

[0031] Control frequency: In straight-line scenarios, the control frequency must be greater than or equal to 5Hz to ensure correction is completed before the deviation exceeds the threshold; in curved scenarios, the control frequency must be greater than or equal to 10Hz to match the dynamic compensation for train steering. The communication protocol supports millisecond-level response, and the frequency is determined by the host computer to meet the requirements.

[0032] Dynamic Response: Required angular velocity for entering curves: 1.3°–4.3° / s (corresponding to curve radii of 300–1000m). The equipment's angular velocity range is 0.01°–20° / s, significantly higher than the maximum requirement. Existing gimbal equipment meets the project requirements in terms of control accuracy, response frequency, and dynamic tracking capability, ensuring stable tracking of the target within ±0.5° of the center of the field of view.

[0033] The pan-tilt base has two outgoing interfaces: a power interface and a video control interface, both of which connect to the equipment box. The equipment box houses the communication control unit and power supply unit for the entire device, and has four outgoing interfaces responsible for supplying 30V DC voltage and communication transmission. Only one network cable is used externally; all communication will be transmitted through the network port. The equipment box integrates a serial port pass-through module.

[0034] The train inspection system employs an Ethernet-based serial port pass-through architecture to achieve precise turntable control: the host computer sends standard Pelco-D protocol commands to the turntable via an RJ45 network port. These commands are encapsulated at the TCP / IP network layer, and then the turntable's built-in serial port pass-through module performs data reconstruction and protocol conversion. The pass-through module restores the network data packets to their original serial signals (RS485 level), forwarding them directly to the internal PTZ controller without any parsing, thus forming a transparent transmission channel between the network port and the serial bus.

[0035] The train detection system employs a dual-layer control architecture of preset position focusing and real-time dynamic angle correction to ensure that the camera continuously locks onto the track target while the train is running at high speed. Preset position control first establishes a track node-preset position mapping table by pre-setting the gimbal focusing angle at key curve locations (entry / exit points) based on a priori electronic map. During train operation, the system acquires real-time train GPS / IMU pose data, locates the current track node through map matching, calls the corresponding preset position coordinates, and sends Pelco-D commands to drive the gimbal to quickly jump to the preset viewing angle.

[0036] S2. Based on the train's pose information and the pre-stored electronic track map, determine the target physical track section ahead of the train at the current acquisition time, and project the target physical track section onto the visible light image, low-light image, and infrared thermal radiation image respectively to obtain the visible light ROI, low-light ROI, and infrared ROI; extract the color texture features of the visible light ROI region, extract the edge contour features of the low-light ROI region, and extract the temperature distribution features of the infrared ROI region; calculate the average brightness value of the visible light ROI region, and determine the confidence level of the visible light ROI region based on the average brightness value; process the low-light image into a gradient magnitude map, statistically analyze the information entropy and local noise intensity of the gradient magnitude map, and determine the confidence level of the low-light ROI based on the information entropy and local noise intensity; calculate the global standard deviation of the temperature values ​​of all pixels in the infrared ROI region, and determine the confidence level of the infrared ROI based on the global standard deviation.

[0037] The pose information is matched with the high-precision electronic track map pre-stored in the vehicle to determine the target track section ahead of the train at the current moment (e.g., the section between 50 meters and 1000 meters ahead). Using the intrinsic and extrinsic parameters of each camera (obtained through calibration) and the pose, the three-dimensional coordinates of the target track section are mapped onto the two-dimensional image planes of the visible light, low light, and infrared images through perspective projection transformation, thereby obtaining three strictly corresponding ROI regions in space.

[0038] Within the visible light ROI region, the color texture feature vector is extracted using the Local Binary Pattern (LBP) algorithm and color histogram. Within the low-light ROI region, the edge image is extracted using the Canny edge detection algorithm, and its shape and contour feature vector is extracted based on the gradient orientation histogram. Within the infrared ROI region, temperature statistical moments are calculated and local extrema are detected to extract its temperature distribution feature vector.

[0039] Calculate the average brightness pixel value L_mean of the grayscale image of the visible light ROI region. Substitute L_mean into the following double sigmoid function to calculate the confidence score C1:

[0040] Where θ1 is the low-light threshold, experimentally calibrated to 30 (corresponding to near-total black); θ2 is the overexposure threshold, experimentally calibrated to 220 (corresponding to severe overexposure); k1=k2=0.05 are the curve sensitivity coefficients. This function ensures that when L_mean is between 30 and 220, The value is greater than 0.95; when L_mean is less than 10 or greater than 240, The value is less than 0.05, thus suppressing the feature weights when the lighting is poor.

[0041] Denoising preprocessing is performed on the low-light ROI image, and its gradient magnitude map M is calculated using the Sobel operator. The histogram distribution of M is statistically analyzed, and its information entropy is calculated. Simultaneously, bilateral filtering or guided filtering is applied to the low-light ROI image to suppress random noise while preserving edges. The local variance of flat areas in the image is calculated as the noise intensity estimate N. The information entropy is corrected using the noise intensity to obtain the effective texture entropy. The effective texture confidence of the low-light ROI is then calculated. Calculated by the following formula:

[0042] ); in, For clarity, the entropy value is used as a reference. This is the noise penalty coefficient. This formula indicates that only when the information in the image originates from real texture (…) (High) rather than noise intensity estimate A low entropy value results in a high confidence score; otherwise, even if the entropy value is large, the confidence score will decrease after subtracting the noise penalty, thus reducing false positives. The confidence score increases with the increase of texture entropy, and reaches its maximum saturation value when the texture entropy reaches or exceeds the reference entropy value.

[0043] Calculate the global standard deviation of the temperature values ​​of all pixels within the infrared ROI region. This index reflects the dispersion of heat distribution within a region. Infrared ROI thermal significance confidence level. The calculation is as follows:

[0044] ; This formula shows that when thermal equilibrium occurs (the target and background are at the same temperature), the temperature distribution within the ROI is uniform, and the standard deviation is... The confidence level is calculated as the value approaches 0. When the infrared weight approaches zero, the system automatically reduces the infrared weight; when a significant thermal target (whether a high-temperature heat source or a low-temperature cold source) is present, the temperature fluctuation within the region increases. Significantly increased, confidence level It rapidly approaches 1, ensuring effective capture of various thermal anomalies.

[0045] S3. Determine the weights of color texture features, edge contour features, and temperature distribution features based on the confidence levels of the visible light ROI, low light ROI, and infrared ROI, respectively; fuse the color texture features, edge contour features, and temperature distribution features according to the weights to obtain weighted fused features; input the weighted fused features into a pre-trained classifier to complete the detection of foreign objects on railway tracks.

[0046] The three confidence levels are Softmax normalized to obtain the fusion weights [W1, W2, W3]. The feature vectors are then weighted according to the weights and concatenated to form the final weighted fusion feature vector.

[0047] The classifier is trained using feature samples generated through weighted fusion of features and their corresponding foreign object annotations. The weighted fusion feature vector is input into a pre-trained Support Vector Machine (SVM) classifier, and trained in conjunction with manually labeled foreign objects (such as "pedestrian", "animal", "falling rock", "background"). The classifier outputs the category of the foreign object, thus completing the detection.

[0048] This invention designs confidence quantification evaluation methods that are highly matched to the imaging mechanisms of three sensors with very different physical properties, thereby providing accurate, reliable and adaptive dynamic weighting basis for subsequent multimodal intelligent fusion.

[0049] To address the high dependence of visible light cameras on ambient light, this invention introduces a dual-threshold evaluation strategy. This method not only sets a low-light threshold to prevent images from being too dark, but also defines a high-light threshold to prevent images from being overexposed, thus accurately defining the camera's optimal operating range. Its core advantage lies in the smooth and continuous change in confidence output around the thresholds, completely avoiding the weight jumps and result jitter caused by traditional binary switching control. This allows the system to achieve an elegant transition in weight fusion under gradually changing light conditions such as sunrise / sunset and tunnel travel, significantly improving output stability. Furthermore, the two thresholds have clear physical meanings and can be easily calibrated using actual line data, ensuring the algorithm's good adaptability to different operating environments.

[0050] For low-light cameras designed to improve visibility in dark environments, this invention abandons simple brightness analysis and instead evaluates the effective information content of the image. This invention introduces a signal-to-noise ratio (SNR) evaluation mechanism that, while statistically analyzing the image gradient entropy, subtracts the background noise introduced by high gain. Its superiority lies in its ability to accurately distinguish between effective texture and random noise. Even when supplemental lighting causes localized overbrightness but overall blurring, or in scenes filled with uniform noise, this method accurately identifies the degradation in image quality without misjudging high reliability. This provides a direct and robust metric for evaluating the image softening and detail loss issues of low-light cameras in extremely low light conditions.

[0051] Since infrared thermal imaging is essentially temperature sensing, the core of this invention is to assess the significance of thermal contrast between the target and the background. It not only calculates the temperature difference but also innovatively introduces the standard deviation of the background temperature field as a normalization factor. This means that the same temperature difference will receive a high significance score in a uniformly temperatureed background, while it will be suppressed in a temperature-chaotic environment. The method is designed to respond to absolute temperature differences; the confidence level monotonically increases as long as a significant thermal anomaly exists. This perfectly aligns with the physical intuition that more significant thermal features lead to more reliable detection, thus avoiding missed detections of low-temperature targets.

[0052] Three confidence assessment schemes, addressing the core reliability challenges faced by different sensors in complex railway environments from three dimensions—light adaptability, texture clarity, and thermal contrast significance—collectively constitute an all-weather "sensor health monitoring system." This system ensures that, under any harsh operating conditions, it can automatically and smoothly allocate decision weights to the most reliable one or a few sensors based on real-time assessment results. For example, the system primarily uses visible light during the day, seamlessly switches to low-light and infrared at night, and automatically reduces the weight of visible light when encountering strong glare at tunnel entrances. This ultimately realizes a truly all-weather, fully adaptive railway foreign object detection system capable of intelligently sensing the environment and autonomously optimizing decisions.

[0053] After obtaining image samples, to address the problem of field-of-view deviation under curved conditions, which leads to shortened detection distance or target loss, this invention provides a train forward-facing multi-sensor fusion detection system based on low-light sensing. This system aims to solve the following main technical problems:

[0054] This invention addresses the issue of monitoring areas deviating from the track due to a fixed viewing angle when trains are operating on curves. To address the shortcomings of existing fixed installation methods where the field of view does not align with the track during curves, this invention aims to provide a sensing platform with responsive control capabilities. This platform can adjust the viewing axis direction in real time according to the train's trajectory, ensuring that the sensor's field of view always covers the effective track area in front of the train, eliminating blind spots in curves. Furthermore, this invention resolves the issues of spatiotemporal asynchrony and environmental adaptability during multi-sensor data fusion. Addressing the problem of inconsistent data acquisition times from multiple sensors during high-speed movement leading to subsequent fusion algorithm failure, this invention aims to provide an integrated solution with a high-precision time synchronization mechanism and a highly reliable protection design. This ensures strict alignment of multimodal data in the time dimension and guarantees stable operation of the equipment under high vibration and strong impact environments caused by trains.

[0055] like Figure 3As shown, real-time angle correction, during train operation, firstly, calculates the target deflection angle Δα = θ – ψ based on the real-time forward angle θ of the track 800m ahead of the train and the train's heading angle ψ. Next, a first-order low-pass filter is applied to Δα to suppress high-frequency vibration interference. Finally, the filtered angle increment is encapsulated as a Pelco-D dynamic control command and sent to the gimbal via the RS485 bus, achieving sub-second (≤100ms) continuous attitude calibration.

[0056] like Figures 4 to 6 As shown, in daytime scenes, low-light cameras exhibit visual imaging characteristics similar to visible light cameras. Infrared cameras, however, highlight the target outlines of railway tracks and people along the roadside through differences in thermal radiation. Specifically, the infrared image shows that the grayscale value of the right track is significantly higher than that of the left track. Analysis reveals this is due to the asymmetrical load of trains traveling in both directions: the right track is the direction of heavy-load trains (coal trains), where the greater axle load intensifies the frictional heating effect between the wheels and rails, causing the rail temperature to rise; while the left track is the direction of empty trains, resulting in a lower temperature rise. This temperature difference characteristic allows infrared cameras to achieve high-sensitivity detection of heavily loaded tracks based on differences in thermal characteristics.

[0057] like Figures 7 to 12 As shown, the image comparison of the same scene during the train's entry and exit from the tunnel is illustrated. First, the infrared thermal imaging camera exhibits strong environmental adaptability, maintaining a high degree of consistency in its imaging effect both inside and outside the tunnel, largely unaffected by sudden changes in illumination. However, limited by the spatial resolution and imaging principle of the infrared sensor, in tunnel scene 2, when a person appears ahead of the track, the thermal radiation characteristics of the person's head overlap with the thermal characteristics of the track background (thermal crossover), resulting in extremely low contrast between the target and the background. The system struggles to effectively separate and identify the person from the infrared image. Second, the visible light camera is most significantly affected by ambient light. At the moment the train enters the tunnel, the ambient illumination drops drastically, causing the visible light image to lose valuable information. At the moment of exiting the tunnel, the strong backlighting at the exit causes severe overexposure, leading to image saturation distortion and an inability to perceive the road conditions ahead.

[0058] In contrast, the low-light camera demonstrated excellent wide dynamic range and low-light adaptability. Although the image briefly darkened upon entering a tunnel, after approximately 1 to 2 seconds of automatic exposure and gain adjustment, it was able to quickly restore clear detection of the scene ahead by making full use of the train's forward illumination (roof lighting). Furthermore, in the high-dynamic scene of exiting a tunnel, the low-light camera was less affected by overexposure and retained more image detail.

[0059] In summary, while low-light and visible-light cameras can effectively distinguish human silhouettes, single sensors have limitations in specific scenarios. Therefore, this invention utilizes multi-sensor collaborative sensing and fusion to effectively complement the advantages of each modality, thereby enabling accurate target detection and identification in complex lighting conditions such as tunnel entrances and exits.

[0060] like Figures 13 to 15 As shown, a comparison of multi-sensor imaging in low-light conditions at night is presented. First, the low-light camera exhibits certain limitations when dealing with complex nighttime lighting environments. Due to interference from high-dynamic, strong light sources in the scene (such as headlights of passing trains and roadside lights), bright areas in the images show some blooming / smearing. Nevertheless, experiments show that even under extremely low illumination conditions of 0.003 Lux, the low-light camera does not completely fail, retaining effective detection capabilities for forward-moving railway tracks and providing basic environmental visual information. In stark contrast, the high-resolution visible light camera, limited by its photosensitivity, is almost unable to acquire effective image signals in poorly lit nighttime scenes, effectively rendering it functionally ineffective. Meanwhile, the infrared thermal imaging camera demonstrates excellent illumination-independent characteristics; its images remain stable and unaffected by low-light conditions and strong light sources at night.

[0061] Based on the above comparison Figures 4-15 Based on comparative test data under different typical railway operating conditions, this invention draws the following experimental conclusions: First, the limitations of single-modal sensors are verified: Visible light cameras only have advantages in well-lit daytime environments. At night (below 0.003 Lux) and in scenarios with drastic light changes at tunnel entrances and exits, they are prone to imaging failure (complete darkness) or information loss (overexposure), failing to meet all-weather detection requirements. Infrared thermal imaging cameras possess strong light independence and environmental stability, effectively detecting targets with temperature difference characteristics (such as tracks heated by friction from heavy-load trains, biological heat sources). However, in thermal equilibrium states (such as when human bodies overlap with background thermal features in tunnels), they are prone to missed detections due to lack of contrast and cannot provide necessary texture details. Low-light cameras fill the blind spots of visible light cameras in extremely low-light environments, capable of restoring scene textures using weak light. Although there is a slight blooming effect under direct strong light at night, their ability to dynamically adjust lighting and capture details in low-light environments makes them an indispensable source of visual perception in nighttime and tunnel scenarios.

[0062] Secondly, the multi-sensor servo detection system integrating low-light, visible light, and infrared cameras proposed in this invention achieves all-weather, all-scene coverage of the railway's forward environment through the complementary advantages of data from each modality. It utilizes the low-light camera to supplement nighttime texture information, the infrared camera to capture heat source targets, and the visible light camera to acquire high-precision details during the day. This effectively overcomes the perception bottleneck of a single sensor under extreme conditions such as strong backlight, extremely low illumination, and low thermal contrast. Even when the performance of a single sensor degrades due to environmental interference (such as strong light interference for the low-light camera or thermal cross-interference for the infrared camera), the system can still acquire effective information through other modalities, significantly reducing the false alarm and missed detection rates for foreign object intrusion detection.

[0063] In summary, by introducing low-light sensing technology and combining it with a multimodal fusion scheme, this invention fundamentally solves the sensing challenges of existing technologies in complex lighting and dynamic environments, providing a more reliable guarantee for the safe forward operation of trains.

[0064] Based on the same inventive concept, this invention also provides a foreign object detection system for railway scenarios, comprising: The data acquisition module is used to simultaneously acquire visible light images, low-light images, and infrared thermal radiation images in front of the train, and record the train's position and posture information at the time of acquisition.

[0065] The calculation module is used to determine the target physical track segment ahead of the train at the current acquisition time based on the train pose information and the pre-stored electronic track map, and to project the target physical track segment onto the visible light image, low-light image, and infrared thermal radiation image respectively to obtain the visible light ROI, low-light ROI, and infrared ROI; extract the color texture features of the visible light ROI region, extract the edge contour features of the low-light ROI region, and extract the temperature distribution features of the infrared ROI region; calculate the average brightness value of the visible light ROI region, and determine the confidence level of the visible light ROI region based on the average brightness value; process the low-light image into a gradient magnitude map, and statistically analyze the information entropy and local noise intensity of the gradient magnitude map, and determine the confidence level of the low-light ROI based on the information entropy and local noise intensity; calculate the global standard deviation of the temperature values ​​of all pixels in the infrared ROI region, and determine the confidence level of the infrared ROI based on the global standard deviation.

[0066] The detection module is used to determine the weights of color texture features, edge contour features, and temperature distribution features based on the confidence levels of the visible light ROI, low light ROI, and infrared ROI, respectively; to fuse the color texture features, edge contour features, and temperature distribution features according to the weights to obtain weighted fused features; and to input the weighted fused features into a pre-trained classifier to complete the detection of foreign objects on the railway track.

[0067] This invention also provides a computer device. At the hardware level, this computer device includes a processor, an internal bus, a network interface, memory, and non-volatile memory, and may also include other hardware required for various operations. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement the foreign object detection method for railway scenarios described above.

[0068] The present invention also provides a computer-readable storage medium storing a computer program that can be used to execute the foreign object detection method in the railway scenario described above.

[0069] Specific limitations regarding the computational system for foreign object detection in railway scenarios can be found in the above section on limitations for foreign object detection methods in railway scenarios, and will not be repeated here. Each module in the aforementioned foreign object detection system for railway scenarios can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0070] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification. Furthermore, the above embodiments only illustrate several implementation methods of this application, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for detecting foreign objects in a railway scenario, characterized in that, include: Simultaneously acquire visible light images, low-light images, and infrared thermal radiation images in front of the train, and record the train's position and attitude information at the time of acquisition; Based on the train's pose information and the pre-stored electronic track map, the target physical track section in front of the train at the current acquisition time is determined, and the target physical track section is projected onto the visible light image, low light image, and infrared thermal radiation image respectively to obtain the visible light ROI, low light ROI, and infrared ROI; the color texture features of the visible light ROI region are extracted, the edge contour features of the low light ROI region are extracted, and the temperature distribution features of the infrared ROI region are extracted. Calculate the average brightness value of the visible light ROI region, and determine the confidence level of the visible light ROI region based on the average brightness value; process the low-light image into a gradient magnitude map, and statistically analyze the information entropy and local noise intensity of the gradient magnitude map, and determine the confidence level of the low-light ROI based on the information entropy and local noise intensity; calculate the global standard deviation of the temperature values ​​of all pixels in the infrared ROI region, and determine the confidence level of the infrared ROI based on the global standard deviation. The weights of color texture features, edge contour features, and temperature distribution features are determined by the confidence levels of the visible light ROI, low light ROI, and infrared ROI, respectively. The color texture features, edge contour features, and temperature distribution features are fused according to the weights to obtain weighted fused features. The weighted fused features are then input into a pre-trained classifier to complete the detection of foreign objects on railway tracks.

2. The foreign object detection method in a railway scenario according to claim 1, characterized in that, The average brightness value of the visible light ROI region is calculated. Based on the average brightness value, the confidence level of the average brightness value of the visible light ROI region is calculated using a double Sigmoid function with low light threshold and overexposure threshold as key parameters. The dual Sigmoid function outputs a confidence value close to 1 when the brightness is between the low light threshold and the overexposure threshold, and outputs a confidence value close to 0 when the brightness is below the low light threshold or above the overexposure threshold. The obtained confidence value is the confidence value of the visible light ROI region.

3. The foreign object detection method in a railway scenario according to claim 1, characterized in that, The low-light image is processed into a gradient magnitude map using the Sobel gradient operator, and the local variance of the flat region of the low-light image is calculated to obtain the noise intensity. The information entropy of the gradient magnitude map is statistically analyzed, and the information entropy is corrected using the noise intensity to obtain the texture entropy. The texture entropy is compared with a preset sharpness reference entropy value, and the confidence level of the low-light ROI is determined according to the ratio of texture entropy to reference entropy value. The confidence level increases with the increase of texture entropy, and the confidence level reaches its maximum saturation value when the texture entropy reaches or exceeds the reference entropy value.

4. The foreign object detection method in a railway scenario according to claim 1, characterized in that, Calculate the global standard deviation of the temperature values ​​of all pixels within the infrared ROI region, and determine the confidence level of the infrared ROI based on the global standard deviation. Specifically, this includes: calculating the standard deviation of the temperature values ​​of all pixels within the infrared ROI region, establishing a monotonically increasing mapping function based on the standard deviation, and calculating the confidence level of the infrared ROI; the mapping function is configured such that: when the standard deviation is larger, a higher confidence level is output; when the standard deviation approaches zero, the lowest confidence level is output.

5. The foreign object detection method in a railway scenario according to claim 1, characterized in that, Before simultaneously acquiring visible light images, low-light images, and infrared thermal radiation images in front of the train, the process also includes: visual calibration of the devices acquiring the three types of images. Specifically, based on the train's pose information and a pre-stored electronic track map, the angle deviation between the extension direction of the track in front of the train and the train's current heading is obtained; according to the angle deviation, the pan-tilt unit carrying the devices for acquiring the visible light images, low-light images, and infrared thermal radiation images is controlled to rotate, so that the line of sight of the acquisition devices is aligned with the track in front, thereby completing the visual calibration of the devices for acquiring the three types of images.

6. The foreign object detection method in a railway scenario according to claim 5, characterized in that, The calculation of the angle deviation specifically includes: obtaining the coordinates of a track point at a predetermined distance ahead from the electronic track map based on the current position of the train, and calculating the azimuth angle of the track point relative to the current position of the train; the angle deviation is the difference between the azimuth angle and the train's heading angle.

7. The foreign object detection method in a railway scenario according to claim 5, characterized in that, The line-of-sight alignment specifically includes: based on the electronic track map, presetting the gimbal angle at key locations on track curves and establishing a track node-preset position mapping table; during train operation, according to the real-time position of the train, calling the corresponding preset position coordinates from the track node-preset position mapping table and driving the gimbal to quickly jump; when the gimbal points to the preset position, calculating continuous angle corrections based on the real-time relative position of the train and the track, and driving the gimbal to perform sub-second dynamic tracking to achieve line-of-sight alignment.

8. A foreign object detection system for railway scenarios, characterized in that, include: The data acquisition module is used to simultaneously acquire visible light images, low-light images, and infrared thermal radiation images in front of the train, and record the train's position and posture information at the time of acquisition. The calculation module is used to determine the target physical track segment in front of the train at the current acquisition time based on the train pose information and the pre-stored electronic track map, and to project the target physical track segment onto the visible light image, low light image and infrared thermal radiation image respectively to obtain the visible light ROI, low light ROI and infrared ROI; extract the color texture features of the visible light ROI region, extract the edge contour features of the low light ROI region, and extract the temperature distribution features of the infrared ROI region; Calculate the average brightness value of the visible light ROI region, and determine the confidence level of the visible light ROI region based on the average brightness value; process the low-light image into a gradient magnitude map, and statistically analyze the information entropy and local noise intensity of the gradient magnitude map, and determine the confidence level of the low-light ROI based on the information entropy and local noise intensity; calculate the global standard deviation of the temperature values ​​of all pixels in the infrared ROI region, and determine the confidence level of the infrared ROI based on the global standard deviation. The detection module is used to determine the weights of color texture features, edge contour features, and temperature distribution features based on the confidence levels of the visible light ROI, low light ROI, and infrared ROI, respectively; to fuse the color texture features, edge contour features, and temperature distribution features according to the weights to obtain weighted fused features; and to input the weighted fused features into a pre-trained classifier to complete the detection of foreign objects on the railway track.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is loaded by the processor, it is able to perform the steps of the method according to any one of claims 1 to 7.

Images