Railway track fastener defect detection method and system based on multi-modal image fusion

Abstract

The application provides a railway track fastener defect detection method and system based on multi-modal image fusion, and relates to the technical field of railway defect detection. The method comprises the following steps: an edge side device acquires a first image set composed of multiple first images with different exposure degrees, and generates a high-quality second image through polarization correction; a target edge device is elected through multi-dimensional evaluation, and the convergence upload of the second image set is realized; a cloud server side integrates light source correction and vehicle body state correction to perform fusion enhancement on the image set, and obtains an optimized target image; finally, a defect detection model is used to output a defect recognition result. The application aims to improve the problem that the existing fastener defect detection scheme has low defect recognition accuracy at night.

Description

Technical Field

[0001] This application relates to the field of railway defect detection technology, and in particular to a method and system for detecting defects in railway track fasteners based on multimodal image fusion. Background Technology

[0002] Railway track fasteners are key connecting components in railway track structures, primarily used to securely fix rails to sleepers, maintain track gauge, bear train loads, and transmit forces to the sleepers and ballast bed. With increasing train speeds and loads, the working environment for fasteners is becoming increasingly harsh, leading to a higher risk of fatigue damage, mechanical aging, and external impact damage.

[0003] To minimize disruption to normal transportation, defect inspection of railway track fasteners is typically conducted at night. However, nighttime inspections are subject to various limitations, leading to a lower accuracy rate in identifying fastener defects. Summary of the Invention

[0004] This application provides a method and system for detecting defects in railway track fasteners based on multimodal image fusion. The embodiments of this application adopt the following technical solutions:

[0005] In a first aspect, embodiments of this application provide a method for detecting defects in railway track fasteners based on multimodal image fusion. The method includes: an edge-side device acquiring a first image set of each railway track fastener to be detected acquired by an image acquisition device, wherein the first image set includes multiple frames of first images of the railway track fasteners to be detected with different exposures.

[0006] The edge-side device determines the polarization correction coefficient of the first image set and corrects the first image set according to the polarization correction coefficient to obtain a second image of the railway track fastener to be detected. The information entropy and dynamic range of the second image are greater than those of the first image.

[0007] The target edge device is determined from multiple edge devices. The target edge device receives the second image sent by other edge devices and uploads the second image set composed of multiple second images to the cloud server.

[0008] The cloud server obtains the light source correction coefficient and vehicle body state correction coefficient of the second image set, and performs fusion and enhancement processing on the second image set according to the light source correction coefficient and vehicle body state correction coefficient to obtain the target image of each railway track fastener to be detected. The clarity and feature separability of the target image are greater than those of the second image.

[0009] Input the target image of each railway track fastener to be inspected into the preset fastener defect detection model to obtain the defect detection result corresponding to each railway track fastener to be inspected.

[0010] In one possible implementation of the first aspect, the edge-side device determines polarization correction coefficients for the first image set, including:

[0011] The edge-side device acquires multiple polarization angles for each first image in the first image set;

[0012] The degree of polarization of each first image is determined based on multiple polarization angles;

[0013] Based on the matching relationship between the degree of polarization and the preset polarization degree range threshold, the polarization correction coefficient corresponding to each first image is determined.

[0014] In one possible implementation of the first aspect, correcting a first image set according to a polarization correction coefficient to obtain a second image of the railway track fastener to be detected includes:

[0015] Based on the polarization correction coefficient, multiple frames of first images with different exposures in the first image set are weighted and fused to generate a preliminary fused image.

[0016] Polarization components are separated from the initial fused image to extract and suppress the specular reflection component, while the diffuse reflection component carrying details of the fastener surface is enhanced.

[0017] The enhanced image undergoes adaptive contrast adjustment and detail enhancement processing, and the output is a second image of the railway track fastener to be detected. This second image is superior to any single frame of the first image in terms of information entropy and dynamic range.

[0018] In one possible implementation of the first aspect, determining a target edge-side device from a plurality of edge-side devices includes:

[0019] Calculate the network connectivity quality score for each candidate edge device based on its location information;

[0020] Based on the data value weight of the second image to be uploaded by each candidate edge device and its own available computing resources, calculate the node comprehensive capability score of each candidate edge device.

[0021] Based on the network connectivity quality score and the node comprehensive capability score, the comprehensive evaluation score of each candidate edge device is calculated by weighted summation.

[0022] The candidate edge device with the highest comprehensive evaluation score will be selected as the target edge device.

[0023] In one possible implementation of the first aspect, a network connectivity quality score for each candidate edge-side device is calculated based on the location information of each edge-side device, including:

[0024] Based on the location information, determine the first distance between the candidate edge-side device and other candidate edge-side devices;

[0025] Based on the location information, determine the second distance between the candidate edge device and the gateway node;

[0026] Based on the first distance, the second distance, and the corresponding weighting coefficients, the network connection quality score of each candidate edge device is determined, wherein the first distance and the second distance are negatively correlated with the network connection quality score.

[0027] In one possible implementation of the first aspect, the cloud server obtains the light source correction coefficients for the second image set, including:

[0028] The two second images with the closest timestamps in the second image set are selected as the first reference image and the second reference image.

[0029] The pixel movement speed is determined based on the first and second reference images;

[0030] The type of interfering light source for each second image is determined based on the pixel movement speed and the exposure brightness of each second image;

[0031] When the interference light source is a dynamic light source, the dynamic light source correction coefficient of the second image set is determined based on the light source motion model and the image sequence.

[0032] When the interference light source is a static light source, the static light source correction coefficient of the second image set is determined based on the difference between the average brightness field of the multi-frame images and the calibration brightness field.

[0033] In one possible implementation of the first aspect, determining the type of interfering light source for each second image based on the pixel movement speed and the exposure brightness of each second image includes:

[0034] Compare the pixel movement speed with a preset speed threshold;

[0035] Analyze the brightness distribution, shape stability, and movement trajectory of the highlighted areas in the second image in the image sequence;

[0036] If the pixel movement speed is lower than the speed threshold, and the brightness distribution of the bright area is uniform, the shape is stable, and the position in the image sequence is fixed, the interference light source type of the second image is determined to be a static light source.

[0037] If the pixel moving speed is higher than or equal to the speed threshold, and the brightness distribution of the bright area is uneven, the shape is unstable, and the position in the image sequence changes discontinuously, the interference light source type of the second image is determined to be a dynamic light source.

[0038] In one possible implementation of the first aspect, the edge-side device is mounted on the detection vehicle, and the cloud server obtains the vehicle body state correction coefficient, including:

[0039] Acquire multi-dimensional operational data of the testing vehicle and the operational trajectory of edge-side devices;

[0040] Based on the correspondence between the operating trajectory of the edge-side equipment and the multi-dimensional operating data of the detection vehicle, the vehicle body state correction coefficient is determined.

[0041] In one possible implementation of the first aspect, a target image of each railway track fastener to be inspected is input into a preset fastener defect detection model to obtain a defect detection result corresponding to each railway track fastener to be inspected, including:

[0042] Input the target image into the fastener defect detection model to obtain evaluation scores for different fastener defect types in the target image;

[0043] The type of fastener defect with the highest evaluation score is identified as the fastener defect detection result for the railway track fastener to be inspected.

[0044] Secondly, this application provides another railway track fastener defect detection system based on multimodal image fusion, the system comprising:

[0045] The first image processing module is used for the edge-side device to acquire a first image set of each railway track fastener to be detected acquired by the image acquisition device. The first image set includes multiple frames of first images of the railway track fasteners to be detected with different exposures.

[0046] The second image processing module is used by the edge-side device to determine the polarization correction coefficient of the first image set and to correct the first image set according to the polarization correction coefficient to obtain a second image of the railway track fastener to be detected. The information entropy and dynamic range of the second image are greater than those of the first image.

[0047] The data upload module is used to determine the target edge device from multiple edge devices. The target edge device receives the second image sent by other edge devices and uploads the second image set composed of multiple second images to the cloud server.

[0048] The third image processing module is used to obtain the light source correction coefficient and vehicle body state correction coefficient of the second image set on the cloud server side, and to perform fusion and enhancement processing on the second image set according to the light source correction coefficient and vehicle body state correction coefficient to obtain the target image of each railway track fastener to be detected. The clarity and feature separability of the target image are greater than those of the second image.

[0049] The identification module is used to input the target image of each railway track fastener to be inspected into the preset fastener defect detection model in order to obtain the defect detection result corresponding to each railway track fastener to be inspected.

[0050] Thirdly, this application also provides an electronic device, which includes: a memory and one or more processors, the memory being coupled to the processors; wherein the memory stores computer program code, the computer program code including computer instructions, which, when executed by the processor, cause the electronic device to perform the method in any of the possible design embodiments of the first aspect described above.

[0051] Fourthly, this application provides a computer-readable storage medium including computer instructions; when the computer instructions are executed on an electronic device, they cause the electronic device to perform the method described in the first aspect above and any possible design of the above.

[0052] Fifthly, this application provides a computer program product that, when run on an electronic device, causes the electronic device to perform the method described in the first aspect above and any possible design of the above.

[0053] This application provides a method for detecting defects in railway track fasteners based on multimodal image fusion. By constructing a hierarchical and collaborative processing architecture, it significantly improves detection accuracy and system efficiency. Specifically, this application employs polarization imaging and multi-exposure fusion technology at the edge side to effectively suppress specular reflection interference from metal surfaces from a physical optics perspective, generating intermediate images with higher information entropy and a wider dynamic range. By establishing a device election mechanism based on a multi-dimensional evaluation model, load optimization among edge nodes and data transmission efficiency are maximized. At the cloud service layer, a dual correction model integrating light source characteristic analysis and motion state compensation is used to further eliminate the impact of complex lighting conditions and vehicle motion posture on image quality, ultimately obtaining target images with fundamentally enhanced visual clarity and feature separability. This method effectively solves technical problems such as optical interference suppression, data transmission bottlenecks, and motion artifact elimination in nighttime detection environments. Furthermore, through the collaborative processing of edge computing and cloud intelligence, it optimizes the allocation of computing resources and improves the overall system performance, thereby enhancing the monitoring accuracy of fastener defect detection.

[0054] The technical effects of the second to fifth aspects refer to the technical effects of the first aspect and any of its embodiments, and will not be repeated here. Attached Figure Description

[0055] Figure 1 A schematic diagram illustrating the steps of a railway track fastener defect detection method based on multimodal image fusion provided in this application embodiment;

[0056] Figure 2 A schematic diagram of the functional modules of a railway track fastener defect detection system based on multimodal image fusion provided in an embodiment of this application;

[0057] Figure 3 A schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0058] The terminology used in the following embodiments is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” and “this” are intended to also include expressions such as “one or more,” unless the context clearly indicates otherwise. It should also be understood that in the following embodiments of this application, “at least one” and “one or more” refer to one or more (including two). The character “ / ” generally indicates that the preceding and following objects are in an “or” relationship.

[0059] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0060] Hereinafter, the terms "first," "second," etc., are used for descriptive convenience only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "multiple" means two or more. For example, multiple processing units refer to two or more processing units.

[0061] Furthermore, in the embodiments of this application, "upper," "lower," "left," and "right" are not limited to the orientation of the components schematically placed in the accompanying drawings. It should be understood that these directional terms can be relative concepts, used for relative description and clarification, and can change accordingly depending on the orientation of the components in the accompanying drawings.

[0062] In the embodiments of this application, unless otherwise explicitly specified and limited, the term "connection" should be interpreted broadly. For example, "connection" can be a fixed connection, a detachable connection, or an integral part; it can be a direct connection or an indirect connection through an intermediate medium. In addition, the term "electrical connection" can be a direct electrical connection or an indirect electrical connection through an intermediate medium.

[0063] In this application, the term "module" typically refers to a logically divided functional structure. A "module" can be implemented purely in hardware, or a combination of hardware and software. In this application, "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, B existing alone, or both A and B existing simultaneously.

[0064] In this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being better or more advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0065] Currently, when using models for defect detection in railway track fasteners, the focus is primarily on optimizing the model structure and improving training accuracy. This includes employing deeper neural networks, larger datasets, or more complex feature extraction algorithms to enhance detection accuracy and generalization ability. However, this model-centric optimization approach has gradually reached its marginal effect in improving overall detection performance in real-world scenarios, and it fails to effectively address recognition errors caused by unstable image acquisition conditions.

[0066] Specifically, issues such as image blurring, lighting variations, occlusion, and viewing angle deviations often lead to insufficient original image quality, making it difficult for even high-precision recognition models to consistently output accurate results. Therefore, simply relying on model structure optimization is no longer sufficient to meet the dual requirements of detection accuracy and stability in practical engineering applications.

[0067] Railway track fasteners are critical connecting components in railway systems used to secure rails to sleepers or track slabs. They typically consist of metal components such as elastic clips, bolts, spikes, and track pads. During long-term service, they are susceptible to various forms of damage and defects, including fracture, cracks, plastic deformation, loosening and displacement, corrosion, and loss, due to the combined effects of train cyclic loads, environmental corrosion, temperature stress, and initial installation defects. Railway track fastener defect detection utilizes automated or intelligent technologies to monitor and assess the integrity, tightness, and surface health of the fasteners, aiming to promptly identify and locate these potential safety hazards.

[0068] Implementing accurate and efficient fastener defect detection is of irreplaceable importance for ensuring railway transportation safety. If fastener failure is not addressed in a timely manner, it may cause rail misalignment, track gauge widening, or even catastrophic accidents such as train derailment.

[0069] Currently, automated inspection systems based on computer vision and deep learning algorithms are gradually replacing traditional manual inspections, becoming the mainstream technology in this field. However, due to the high traffic density and busy operations on most railway mainlines during the day, it is difficult to schedule large-scale, long-term manual track inspections or low-speed inspection vehicle operations. Therefore, actual track image acquisition and fastener inspection tasks are usually scheduled during railway maintenance windows at night to minimize disruption to normal transportation.

[0070] However, while nighttime inspection environments avoid the impact of daytime operations, they also face severe technical challenges. When operating, inspection vehicles not only need to turn on their own high-power lighting systems but may also encounter the high beams of oncoming trains, vehicles working on adjacent lines, or other engineering equipment. These strong light sources from multiple directions and heterogeneous dynamics create intense, irregular specular reflections and glare on the surface of metal fasteners, resulting in large overexposed areas and complex light spots in the acquired images. This optical noise severely obscures key defect features such as fine cracks and corrosion on the fastener surface and produces misleading shadows, making it difficult for subsequent image analysis algorithms to extract effective information, directly leading to a decrease in recognition accuracy.

[0071] Secondly, there is the immense pressure that massive amounts of data place on cloud processing. A complete railway inspection system typically includes dozens of image acquisition units deployed across multiple carriages, generating a massive amount of high-resolution image data during a single routine inspection. Uploading all this raw image data directly to the cloud for analysis without processing presents multiple bottlenecks: First, it puts extreme pressure on the bandwidth of the onboard and station communication networks, and the latency and instability of data transmission make it difficult to guarantee real-time detection. Second, cloud servers need to handle a huge amount of raw data decoding, preprocessing, and computation tasks, consuming enormous computing and storage resources, resulting in high overall system operating costs. Third, centralizing all computational tasks in the cloud also increases system response time, making it difficult to meet the real-time requirements for rapid early warning of major defects.

[0072] First, the application scenario of this application is explained. While daytime fastener defect detection has the advantage of good lighting conditions, it conflicts with dense train schedules, making it difficult to implement in practice. However, to minimize the impact on normal railway transportation, detection operations are more often carried out during nighttime railway maintenance windows. Although this solves the problem of train interference, it exposes the detection system to the severe challenge of complex dynamic light source environments. Based on this, the inventive concept of this application is proposed: by deploying multiple edge-side devices with image acquisition and computing capabilities on the detection vehicle, the edge-side devices collaboratively complete multimodal image fusion, dynamic light source characteristic modeling, and polarization adaptive correction, effectively suppressing glare and overexposure caused by external interference light sources such as oncoming trains and nearby operating vehicles, significantly improving the usable quality of fastener images. At the same time, by performing image preprocessing and feature enhancement on the edge side, the amount of data that needs to be uploaded is reduced, improving the problems of complex light source interference at night and cloud processing bottlenecks.

[0073] Reference Figure 1 The present invention provides a method for detecting defects in railway track fasteners based on multimodal image fusion, which may specifically include the following steps:

[0074] S101: The edge-side device acquires the first image set of each railway track fastener to be detected, acquired by the image acquisition device.

[0075] The inspection vehicle in this application is equipped with multiple edge-side devices. Each edge-side device can be deployed in at least one of the following ways: spaced apart along the longitudinal axis of the inspection vehicle to cover continuous track sections; oriented according to different acquisition angles based on the preset position of the fasteners on the sleepers; or installed in a redundant configuration to ensure that images of key sections are acquired without omission. Each edge-side device can dynamically adjust its data acquisition cycle according to the vehicle's speed. Within one data acquisition cycle, each edge-side device synchronously acquires a sequence of fastener images containing multiple sets of different exposure parameters and polarization angles, forming a first image set for subsequent fusion processing. For example, the first image set for each railway track fastener to be inspected may include three first images with different exposures acquired at the same time.

[0076] S102: The edge-side device determines the polarization correction coefficient of the first image set and corrects the first image set according to the polarization correction coefficient to obtain a second image of the railway track fastener to be detected.

[0077] In the embodiments of this application, polarization correction is implemented to fundamentally address the impact of complex multi-source dynamic lighting on fastener inspection at night: First, strong light sources incident from different directions (such as oncoming train headlights and the inspection vehicle's own illumination) will form superimposed highlight areas on the surface of the metal fastener, causing local pixel saturation and complete loss of texture details; second, dynamically changing lighting conditions will cause the same fastener to exhibit drastically different optical characteristics at different times, causing segmentation algorithms based on fixed thresholds to fail; third, shadows and specular reflection patches generated by moving light sources will form interference features such as false edges, seriously misleading the defect identification model. Existing technologies are mostly limited to exposure parameter adjustments or traditional image enhancement algorithms. These methods can only improve global brightness within a limited range, but cannot distinguish between real surface features and transient optical noise. When dealing with multi-source dynamic lighting, they often fall into the dilemma of excessive smoothing resulting in loss of detail or retaining noise to interfere with identification.

[0078] By performing pixel-level registration, polarization state analysis, and multi-scale fusion on multiple frames of the first image set, an initial fused image with higher dynamic range and richer surface details is constructed. Subsequently, based on polarization correction coefficients, the specular reflection component in the initial fused image is selectively suppressed, while the diffuse reflection component is enhanced and reconstructed. This results in a second image that significantly outperforms the original first image in terms of texture sharpness, feature contrast, and noise suppression. This processing fully utilizes the complementary information of multimodal images, effectively overcoming the limitation of information loss in single-frame images under complex nighttime lighting conditions, and providing a reliable data foundation for subsequent defect detection.

[0079] In one feasible implementation, determining the polarization correction coefficients for the first image set includes:

[0080] S1021: The edge-side device acquires multiple polarization angles for each first image in the first image set;

[0081] S1022: Determine the degree of polarization of each first image based on multiple polarization angles;

[0082] S1023: Determine the polarization correction coefficient corresponding to each first image based on the matching relationship between the polarization degree and the preset polarization degree range threshold.

[0083] In the embodiments S1021 to S1023, the edge-side device first acquires multiple first images of the same fastener at different polarization angles using a polarization camera, constructing a complete polarization state observation sequence. Subsequently, the device calculates the polarization degree parameter for each pixel based on the Stokes vector method. This polarization degree parameter precisely quantifies the ratio of specular reflection to diffuse reflection in surface reflection. Finally, the system establishes a dynamic mapping mechanism between polarization degree and preset intervals: high-polarization-degree regions (dominated by specular reflection) are assigned lower correction weights to suppress glare, while low-polarization-degree regions (dominated by diffuse reflection) are assigned higher weights to enhance surface details. This process achieves physical decoupling from complex lighting conditions, providing precise guidance for subsequent image correction.

[0084] As an example, the polarization value of each pixel is first calculated based on the Stokes vector, with a value range of [0,1]. A low polarization threshold T1 = 0.2 and a high polarization threshold T2 = 0.6 are set, dividing the pixel into three feature regions for processing: for the low polarization region (0, 0.2), a correction coefficient of 1.75-1.85 is assigned to enhance surface details; for the medium polarization region (0.2, 0.6), a correction coefficient of 1.2-1.44 is assigned to maintain reflective feature balance; and for the high polarization region (0.6, 1.0), a correction coefficient of 0.3-0.7 is assigned to suppress specular reflection. This piecewise linear mapping mechanism ensures optimal processing results for each feature region, avoids artifacts caused by abrupt boundary changes through smooth transitions, and can adaptively adjust the correction coefficients for parameters of different fastener materials, which is not limited in this application.

[0085] In one feasible implementation, the first image set is corrected according to a polarization correction coefficient to obtain a second image of the railway track fastener to be detected, including:

[0086] S1024: Based on the polarization correction coefficient, perform weighted fusion on multiple frames of first images with different exposures in the first image set to obtain a preliminary fused image;

[0087] S1025: Perform polarization component separation on the preliminary fused image, extract and suppress the specular reflection component, and enhance the diffuse reflection component carrying details of the fastener surface to obtain an intermediate fused image.

[0088] S1026: Perform contrast adaptive adjustment and detail enhancement processing on the intermediate fused image to obtain a second image of the railway track fastener to be detected.

[0089] In the implementations of S1024 to S1026, after obtaining the polarization correction coefficients of each first image in the first image set, adaptive weighted fusion is first performed on multiple frames of first images with different exposures based on the polarization correction coefficients corresponding to each image: low weights are assigned to specular reflection areas to suppress overexposure, and high weights are assigned to diffuse reflection areas to enhance details, generating a preliminary fused image with a significantly expanded dynamic range. Then, the preliminary fused image is decomposed into specular reflection and diffuse reflection components using a polarization physics model, and a nonlinear suppression algorithm is used to eliminate specular reflection interference. At the same time, guided filtering technology is used to enhance the surface micro-features in the diffuse reflection component, obtaining an intermediate fused image with a significantly improved signal-to-noise ratio. Finally, the system enhances the intermediate fused image based on an adaptive histogram equalization method with limited local contrast, and combines a multi-scale detail enhancement algorithm to highlight the defects such as cracks and rust on the fastener surface, ultimately outputting a second image with uniform illumination distribution, high contrast, and rich texture details, providing optimal input data for subsequent defect detection. This complete processing chain is completed at the edge, ensuring a substantial improvement in image quality while effectively reducing the processing burden on the cloud. Because cloud servers do not need to process raw multimodal polarization image data, participate in real-time polarization correction coefficient calculations, or perform multi-exposure image fusion and enhancement operations, all these computationally intensive low-level image processing tasks are completed at the edge. Through this division of labor, cloud servers can fully leverage their core advantages, focusing on critical tasks such as collaborative analysis of multi-device data, iterative execution of complex defect identification algorithms, establishment of long-term condition monitoring and prediction models, and management of the entire line's fastener status database. This division of capabilities between the edge and cloud not only significantly reduces network bandwidth requirements and improves system real-time performance but also optimizes the allocation of computing resources, enabling the cloud to concentrate its limited resources on higher-value data mining and intelligent analysis tasks.

[0090] S103: Determine the target edge device from multiple edge devices. The target edge device receives the second image sent by other edge devices and uploads the second image set composed of multiple second images to the cloud server.

[0091] After each edge device completes polarization correction and multimodal fusion processing of the first image set to generate the corresponding second image, a collaborative upload mechanism between edge devices is needed to achieve efficient aggregation and unified uploading of multi-node detection data and avoid network congestion caused by all edge devices simultaneously establishing connections to the cloud. By comprehensively considering multiple dimensions such as network link quality, device computational load, and data value density, the optimal target device is dynamically selected from multiple edge devices to collect and integrate the second images from other adjacent devices, construct a complete second image set, and then upload it uniformly to the cloud server. This layered aggregation transmission architecture not only alleviates the instantaneous pressure on network bandwidth but also improves the quality and value density of uploaded data by performing preliminary data filtering and integration at the edge.

[0092] Specifically, the target edge-side device is determined from multiple edge-side devices, including:

[0093] S1031: Calculate the network connection quality score for each candidate edge device based on the communication link quality and physical distance between each edge device and the current service gateway node;

[0094] S1032: Calculate the node comprehensive capability score of each candidate edge device based on the data value weight of the second image to be uploaded by each candidate edge device and its own available computing resources;

[0095] S1033: Calculate the comprehensive evaluation score of each candidate edge device by weighted summation based on the network connectivity quality score and the node comprehensive capability score;

[0096] S1034: The candidate edge device with the highest comprehensive evaluation score is selected as the target edge device.

[0097] In the embodiments S1031 to S1034, when determining the target edge device, this application can comprehensively evaluate it from two dimensions: network connection quality score and node comprehensive capability. Through the network connection quality evaluation dimension, the system establishes a quantitative understanding of the device's basic communication capabilities by comprehensively analyzing characteristic parameters such as the physical distance between the device and the gateway, signal transmission quality, and link stability. This evaluation ensures that the selected device possesses reliable data transmission capabilities. In the node comprehensive capability evaluation dimension, by analyzing the feature richness and defect detection value of image content, combined with the device's real-time computing resource status, an accurate evaluation of the device's processing and transmission capabilities is formed. This evaluation dimension ensures that system resources are tilted towards high-value data. By dynamically weighting and fusing the evaluation results of these two dimensions, the system constructs a flexible election model that can adapt to complex field environments. This model can automatically adjust the election strategy according to changes in network status, data characteristics, and device resources, achieving self-organization and self-optimization of the edge device group.

[0098] In one feasible implementation, a network connectivity quality score for each candidate edge-side device is calculated based on the location information of each edge-side device, including:

[0099] S10311: Based on the location information, determine the first distance between the candidate edge-side device and other candidate edge-side devices;

[0100] S10312: Determine the second distance between the candidate edge device and the gateway node based on the location information;

[0101] S10313: Determine the network connection quality score for each candidate edge device based on the first distance, the second distance, and the corresponding weight coefficients, wherein the first distance and the second distance are negatively correlated with the network connection quality score.

[0102] In the implementations of S10311 to S10313, the precise location information of each device can first be obtained through the device's built-in positioning module. Based on this, two key distance parameters are calculated: the first distance between devices reflects the communication cost within the edge device cluster, while the second distance between the device and the gateway characterizes the communication efficiency of data uploading to the infrastructure.

[0103] During the evaluation process, the system uses a negative correlation function to map these two distance parameters to a network connectivity quality score. The first distance assessment reflects the communication efficiency of the device as a data aggregation node; a smaller distance means lower energy consumption and latency for data transmission between devices. The second distance assessment reflects the reliability of the device's connection to the backbone network; a smaller distance indicates a shorter data transmission path and less signal attenuation and interference. By assigning dynamic weight coefficients to these two distance parameters, the system can intelligently adjust the proportion of internal communication and uplink communication in the overall evaluation based on the real-time network topology.

[0104] This evaluation method based on spatial location relationships ensures that the selected target edge devices have both good internal data collection capabilities and efficient uplink communication links, providing a good communication foundation for subsequent data aggregation and uploading tasks.

[0105] As an example, when five candidate edge-side devices (AE) are included, the spatial distribution of the five devices on the detection vehicle and their distance relationship with the gateway can be as follows: Device A: located at the front of the vehicle, with an average distance of 35 meters from other devices and 25 meters from the gateway; Device B: located at the junction of the vehicle and the carriage, with an average distance of 20 meters from other devices and 30 meters from the gateway; Device C: located in the middle of the vehicle and the carriage, with an average distance of 15 meters from other devices and 45 meters from the gateway; Device D: located at the rear of the vehicle and the carriage, with an average distance of 30 meters from other devices and 50 meters from the gateway; Device E: located in a spare position and the average distance of 40 meters from other devices and 35 meters from the gateway.

[0106] In the network connectivity quality score calculation, the weight of the distance between devices is set to 0.6, and the weight of the distance to the gateway is set to 0.4. Based on the principle that distance and score are negatively correlated, an inverse proportional function is used for quantitative calculation: Device B, due to its superior location at the connection point of the carriages, achieves the best balance in the two dimensions of communication distance between devices (20 meters) and gateway connection distance (30 meters), and obtains the highest comprehensive score of 0.82.

[0107] It should be noted that the specific weighting ratios and evaluation criteria can be set according to the actual situation, and this application does not impose any restrictions.

[0108] S104: The cloud server obtains the light source correction coefficient and vehicle body state correction coefficient of the second image set, and performs fusion and enhancement processing on the second image set according to the light source correction coefficient and vehicle body state correction coefficient to obtain the target image of each railway track fastener to be detected.

[0109] After the cloud server acquires the second image set uploaded by the target edge device, it can perform deeper optimization processing on the images to further improve the impact of complex multi-source dynamic lighting on fastener detection at night. Specifically, the cloud server analyzes a multi-frame image sequence in the second image set, first calculating a light source correction coefficient. This coefficient can effectively distinguish and compensate for the uneven lighting caused by dynamic light sources (such as oncoming train headlights) and static light sources (such as fixed lighting on the inspection vehicle) incident from different directions. At the same time, the cloud server calculates a vehicle body state correction coefficient based on the real-time operating parameters of the inspection vehicle, effectively eliminating image blurring and geometric distortion caused by changes in the vehicle's motion state such as vibration and swaying. By applying these two correction coefficients to the fusion and enhancement processing of the second image set, the system can generate target images with uniform illumination distribution and accurate geometric features, thereby overcoming the optical interference caused by the complex nighttime environment on fastener detection at a higher level and providing optimal input data quality assurance for subsequent defect identification.

[0110] Specifically, it includes:

[0111] S1041: Select the two second images with the closest timestamps in the second image set as the first reference image and the second reference image;

[0112] S1042: Determine the pixel movement speed based on the first reference image and the second reference image;

[0113] S1043: Determine the type of interfering light source for each second image based on the pixel movement speed and the exposure brightness of each second image;

[0114] S1044: When the interference light source is a dynamic light source, determine the dynamic light source correction coefficient of the second image set based on the light source motion model and the image sequence.

[0115] S1045: When the interference light source is a static light source, determine the static light source correction coefficient of the second image set based on the difference between the average brightness field and the calibration brightness field of the multi-frame images.

[0116] In the implementations of S1041 to S1045, light source characteristics can be analyzed from two dimensions: pixel movement speed and interference light source type. Specifically, firstly, the precise relative motion speed is calculated through pixel displacement analysis between consecutive frames. This parameter not only reflects the operating state of the detection vehicle but, more importantly, provides a key basis for distinguishing between static and dynamic interference light sources. Based on this, the system combines the exposure brightness distribution characteristics of each frame image to establish a light source classification mechanism based on the collaborative analysis of motion and optical features: when a high-speed moving bright spot is detected that does not match the vehicle's movement trajectory, it is determined to be a dynamic interference light source; when a region with a fixed position but abnormal brightness is identified, it is classified as a static interference light source. This two-dimensional analysis method ensures the accuracy of light source type identification, laying a solid foundation for subsequent targeted correction processing. By adopting differentiated correction strategies for different light source types, effective suppression of multi-source interference in complex nighttime lighting environments is achieved, improving the quality stability of fastener images.

[0117] In one feasible implementation, the type of interfering light source for each second image is determined based on the pixel movement speed and the exposure brightness of each second image, including:

[0118] Compare the pixel movement speed with a preset speed threshold;

[0119] Analyze the brightness distribution, shape stability, and movement trajectory of the highlighted areas in the second image in the image sequence;

[0120] If the pixel movement speed is lower than the speed threshold, and the brightness distribution of the bright area is uniform, the shape is stable, and the position in the image sequence is fixed, the interference light source type of the second image is determined to be a static light source.

[0121] If the pixel moving speed is higher than or equal to the speed threshold, and the brightness distribution of the bright area is uneven, the shape is unstable, and the position in the image sequence changes discontinuously, the interference light source type of the second image is determined to be a dynamic light source.

[0122] In this embodiment, accurately distinguishing between static and dynamic light sources is the prerequisite and foundation for effective image correction. This application achieves this crucial distinction by establishing a multi-dimensional feature analysis system. At the motion feature level, the system analyzes the comparison between pixel movement speed and a preset threshold to identify that static light sources exhibit a stable motion pattern synchronized with the detected vehicle, while dynamic light sources exhibit independent and discontinuous motion characteristics. At the optical feature level, the system analyzes the uniformity of brightness distribution and shape stability in bright areas. Static light sources typically maintain stable light field distribution characteristics, while dynamic light sources exhibit obvious spot flickering and shape changes. At the spatiotemporal behavior level, the system analyzes the changing patterns of light source positions in the image sequence. Static light sources maintain a relatively fixed spatial relationship between consecutive frames, while dynamic light sources show rapid positional transitions. Different types of light source interference require completely different correction strategies. Static light source correction mainly addresses the problem of illumination uniformity compensation, while dynamic light source correction requires specialized handling of time-varying interference such as instantaneous strong light and motion blur. Meanwhile, accurate light source classification enables the system to intelligently allocate computing resources based on interference characteristics, avoiding the use of complex dynamic processing algorithms for static scenes, thereby significantly improving the overall efficiency of the system; more importantly, this distinction ensures the best restoration effect of fastener surface details, providing reliable image quality assurance for subsequent defect detection.

[0123] In one feasible implementation, the cloud server obtains the vehicle body state correction coefficient by: acquiring multi-dimensional operating data of the detection vehicle and the operating trajectory of the edge device, and determining the vehicle body state correction coefficient based on the correspondence between the operating trajectory of the edge device and the multi-dimensional operating data of the detection vehicle.

[0124] In this embodiment, the cloud server can acquire in real-time triaxial acceleration and angular velocity data provided by the vehicle's inertial measurement unit, as well as multi-dimensional operating parameters such as operating speed and steering angle provided by the vehicle control system. Simultaneously, the system analyzes the motion trajectory of the edge-side devices in a continuous image sequence to accurately calculate the actual displacement and vibration of the image acquisition equipment. By performing spatiotemporal alignment and correlation analysis between the vehicle's operating data and the device's motion trajectory, it can accurately distinguish between image distortion caused by vehicle motion and changes in the geometric characteristics of the fasteners themselves. This effectively eliminates the impact of changes in motion states such as vehicle vibration, acceleration / deceleration, and curved driving on image quality, thereby generating a vehicle body state correction coefficient that accurately reflects the true geometric shape of the fasteners.

[0125] S105: Input the target image of each railway track fastener to be inspected into the preset fastener defect detection model to obtain the defect detection result corresponding to each railway track fastener to be inspected.

[0126] In this embodiment, the fastener defect detection model can identify various fastener states, and its output covers multiple defect types such as normal, rotated, reversed, offset, damaged, and missing. By inputting the target image corresponding to each fastener into the fully trained defect detection model, a probability distribution representing the likelihood of each defect type can be obtained.

[0127] As an example, taking the analysis of a target image of a fastener as an example, the probability distribution output by the model might be: normal (0.08), rotated (0.15), offset (0.10), damaged (0.60), and missing (0.07). This probability distribution reflects the model's confidence in the image belonging to each type of defect; a higher value indicates a greater probability of that defect type existing. In this example, the probability value corresponding to the damaged category (0.60) is significantly higher than other types, so the system determines damaged as the final defect detection result for the fastener.

[0128] This application provides a method for detecting defects in railway track fasteners based on multimodal image fusion. By constructing a hierarchical and collaborative processing architecture, it significantly improves detection accuracy and system efficiency. Specifically, this application employs polarization imaging and multi-exposure fusion technology at the edge side to effectively suppress specular reflection interference from metal surfaces from a physical optics perspective, generating intermediate images with higher information entropy and a wider dynamic range. By establishing a device election mechanism based on a multi-dimensional evaluation model, load optimization among edge nodes and data transmission efficiency are maximized. At the cloud service layer, a dual correction model integrating light source characteristic analysis and motion state compensation is used to further eliminate the impact of complex lighting conditions and vehicle motion posture on image quality, ultimately obtaining target images with fundamentally enhanced visual clarity and feature separability. This method effectively solves technical problems such as optical interference suppression, data transmission bottlenecks, and motion artifact elimination in nighttime detection environments. Furthermore, through the collaborative processing of edge computing and cloud intelligence, it optimizes the allocation of computing resources and improves the overall system performance, thereby enhancing the monitoring accuracy of fastener defect detection.

[0129] Secondly, based on the same inventive concept, and referring to... Figure 2 This application illustrates an embodiment of a railway track fastener defect detection system 200 based on multimodal image fusion, the system comprising:

[0130] The first image processing module 201 is used for the edge-side device to acquire a first image set of each railway track fastener to be detected acquired by the image acquisition device. The first image set includes multiple frames of first images of the railway track fasteners to be detected with different exposures.

[0131] The second image processing module 202 is used for the edge-side device to determine the polarization correction coefficient of the first image set, and to correct the first image set according to the polarization correction coefficient, so as to obtain a second image of the railway track fastener to be detected. The information entropy and dynamic range of the second image are greater than those of the first image.

[0132] The data upload module 203 is used to determine the target edge device from multiple edge devices. The target edge device receives the second image sent by other edge devices and uploads the second image set composed of multiple second images to the cloud server.

[0133] The third image processing module 204 is used to obtain the light source correction coefficient and vehicle body state correction coefficient of the second image set on the cloud server side, and perform fusion and enhancement processing on the second image set according to the light source correction coefficient and vehicle body state correction coefficient to obtain the target image of each railway track fastener to be detected. The clarity and feature separability of the target image are greater than those of the second image.

[0134] The recognition module 205 is used to input the target image of each railway track fastener to be inspected into the preset fastener defect detection model in order to obtain the defect detection result corresponding to each railway track fastener to be inspected.

[0135] In other embodiments, an electronic device is provided, which may be a terminal device as described in the above method embodiments, for executing the method steps performed by the terminal in the above method flow. The internal structure diagram of this electronic device may be as follows: Figure 3As shown, the device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The memory of the electronic device may include non-volatile storage medium and internal memory. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computing and control capabilities. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a method for detecting defects in railway track fasteners based on multimodal image fusion. The display unit of the electronic device is used to form a visually visible image and can be a display screen, projection device, or virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the electronic device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the electronic device, or external keyboards, touchpads, or mice, etc.

[0136] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0137] In some embodiments, an electronic device includes a memory, a processor, and a communication interface. The communication interface is used to interact with other devices to send and receive data. For example, in this embodiment, the communication interface may specifically be used to store computer program code, which includes computer instructions. These computer instructions run in the electronic device to implement the method shown in the above-described method embodiments. For example, the memory may include high-speed random access memory (RAM), and may also include non-volatile memory (NVM), such as at least one disk storage device, and may also be a USB flash drive, portable hard drive, read-only memory, disk, or optical disk, etc.

[0138] The processor can be a general-purpose processor, including a Central Processing Unit (CPU), a network processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The processor can also be other general-purpose processors. A general-purpose processor can be a microprocessor or any conventional processor.

[0139] Memory, communication interfaces, and processor communication connections. For example, memory and communication interfaces can connect to the processor via the system bus and communicate with each other. The system bus can be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, an Industry Standard Architecture (ISA) bus, etc. The system bus can be divided into address bus, data bus, control bus, etc. For ease of representation, only one thick line is used in the diagram, but this does not indicate that there is only one bus or one type of bus.

[0140] Alternatively, the memory can be either standalone or integrated with the processor. When the memory is set up independently, it is connected to the processor via the system bus.

[0141] This application also provides a chip for executing instructions, which is used to execute the technical solution of the railway track fastener defect detection method based on multimodal image fusion in the above embodiments.

[0142] This application also provides a computer-readable storage medium storing computer instructions. When these computer instructions are executed by a processor, they are used to implement the technical solution of the railway track fastener defect detection method based on multimodal image fusion in the above embodiments. Specifically, when the computer instructions are executed by a processor, the computer device can execute the technical solution of the railway track fastener defect detection method based on multimodal image fusion provided in the above embodiments.

[0143] This application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium. When the at least one processor executes the computer program, it can implement the technical solution of the railway track fastener defect detection method based on multimodal image fusion provided in the above embodiments.

[0144] The aforementioned computer-readable storage media can be implemented by any type of volatile or non-volatile storage device or a combination thereof. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), etc.

[0145] Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can take many forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). Computer-readable storage media may be any available medium accessible to general-purpose or special-purpose computers.

[0146] An exemplary computer-readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Of course, the computer-readable storage medium can also be a component of the processor. The processor and the computer-readable storage medium can reside in an application-specific integrated circuit (ASIC). Alternatively, the processor and the computer-readable storage medium can exist as discrete components in an electronic control unit or main control device; this application does not limit this.

[0147] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0148] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules in the formula can be selected to implement the solution of this embodiment according to actual needs.

[0149] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The unit composed of the above modules can be implemented in hardware or in the form of hardware plus software functional units.

[0150] The integrated modules described above, implemented as software functional modules, can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods of the various embodiments of this application.

[0151] It should be understood that the steps of the method disclosed in the embodiments of this application can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules in the processor.

[0152] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0153] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification. The above embodiments only illustrate several implementation methods of this application, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of this application's patent. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

[0154] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features in the formula. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A railway track fastener defect detection method based on multi-modal image fusion, characterized in that, The method comprises: An edge-side device acquires a first image set of each railway track fastener to be detected collected by an image collection device, and the first image set comprises multiple frames of first images of the railway track fastener to be detected with different exposure degrees; The edge-side device determines a polarization correction coefficient of the first image set, and corrects the first image set according to the polarization correction coefficient to obtain a second image of the railway track fastener to be detected, and the information entropy and dynamic range of the second image are greater than those of the first image; A target edge-side device is determined from multiple edge-side devices, the target edge-side device receives the second images sent by other edge-side devices, and uploads a second image set composed of multiple second images to a cloud server end; The cloud server end acquires a light source correction coefficient and a vehicle body state correction coefficient of the second image set, and performs fusion and enhancement processing on the second image set according to the light source correction coefficient and the vehicle body state correction coefficient to obtain a target image of each railway track fastener to be detected, and the definition and feature separability of the target image are greater than those of the second image; The target image of each railway track fastener to be detected is input into a preset fastener defect detection model to obtain a defect detection result corresponding to each railway track fastener to be detected; The edge-side device determines the polarization correction coefficient of the first image set, comprising: The edge-side device acquires multiple polarization angles of each first image in the first image set; According to the multiple polarization angles, the polarization degree of each first image is determined; According to the matching relationship between the polarization degree and a preset polarization degree interval threshold, a polarization correction coefficient corresponding to each first image is determined; According to the polarization correction coefficient, the first image set is corrected to obtain a second image of the railway track fastener to be detected, comprising: According to the polarization correction coefficient, multiple frames of first images with different exposure degrees in the first image set are weighted and fused to obtain a preliminary fusion image; Polarization component separation is performed on the preliminary fusion image, and the specular reflection component is extracted and suppressed, and the diffuse reflection component carrying the fastener surface detail information is enhanced to obtain an intermediate fusion image; The intermediate fusion image is subjected to contrast adaptive adjustment and detail enhancement processing to obtain the second image of the railway track fastener to be detected.

2. The method for detecting defects in railway track fastenings based on fusion of multi-modal images according to claim 1, characterized in that, The target edge-side device is determined from multiple edge-side devices, comprising: According to the position information of each candidate edge-side device, the network connection quality score of each candidate edge-side device is calculated; According to the data value weight of the second image to be uploaded by each candidate edge-side device and the available computing resources thereof, the node comprehensive capability score of each candidate edge-side device is calculated; According to the network connection quality score and the node comprehensive capability score, the comprehensive selection score of each candidate edge-side device is calculated by weighted summation; The candidate edge-side device with the highest comprehensive selection score is determined as the target edge-side device.

3. The method for detecting defects in railway track fastenings based on fusion of multi-modal images according to claim 2, characterized in that, The network connection quality score of each candidate edge side device is calculated according to the position information of each edge side device, and the method comprises the steps of: determining a first distance between the candidate edge side device and other candidate edge side devices according to the position information; determining a second distance between the candidate edge side device and the gateway node according to the position information; determining the network connection quality score of each candidate edge side device according to the first distance, the second distance and the corresponding weight coefficient, wherein the first distance and the second distance are in a negative correlation with the network connection quality score.

4. The method for detecting defects in railway track fastenings based on fusion of multi-modal images according to claim 1, characterized in that, The cloud server end obtains the light source correction coefficient of the second image set, comprising: selecting two frames of second images with the closest timestamps in the second image set as a first reference image and a second reference image; determining a pixel movement speed according to the first reference image and the second reference image; determining a type of interference light source of each second image according to the pixel movement speed and the exposure brightness of each second image; in the case that the type of interference light source is a dynamic light source, determining a dynamic light source correction coefficient of the second image set according to a light source motion model and an image sequence; in the case that the type of interference light source is a static light source, determining a static light source correction coefficient of the second image set according to the difference between the average brightness field of multiple images and the calibration brightness field.

5. The method for railway track fastener defect detection based on multi-modal image fusion according to claim 4, characterized in that, The type of interference light source of each second image is determined according to the pixel movement speed and the exposure brightness of each second image, comprising: comparing the pixel movement speed with a preset speed threshold value; analyzing the brightness distribution, shape stability and movement trajectory in the image sequence of a highlight area in the second image; in the case that the pixel movement speed is lower than the speed threshold value, and the brightness distribution of the highlight area is uniform, the shape is stable, and the position in the image sequence is fixed, determining that the type of interference light source of the second image is the static light source; in the case that the pixel movement speed is higher than or equal to the speed threshold value, and the brightness distribution of the highlight area is non-uniform, the shape is unstable, and the position in the image sequence is discontinuously changed, determining that the type of interference light source of the second image is the dynamic light source.

6. The method for railway track fastener defect detection based on multi-modal image fusion according to claim 1, characterized in that, The edge side device is arranged on a detection vehicle, and the cloud server end obtains a vehicle body state correction coefficient, comprising: obtaining multi-dimensional running data of the detection vehicle and a running trajectory of the edge side device; determining the vehicle body state correction coefficient according to the correspondence between the running trajectory of the edge side device and the multi-dimensional running data of the detection vehicle.

7. The method for railway track fastener defect detection based on multi-modal image fusion according to claim 1, characterized in that, The target image of each railway track fastener to be detected is input into a preset fastener defect detection model to obtain a defect detection result corresponding to each railway track fastener to be detected, comprising: inputting the target image into the fastener defect detection model to obtain evaluation scores of different fastener defect types of the target image; determining the fastener defect type with the highest evaluation score as the fastener defect detection result of the railway track fastener to be detected.

8. A railway track fastener defect detection system based on multi-modal image fusion, characterized by, The application discloses a railway track fastener defect detection method and device based on multi-modal image fusion. A first image processing module is configured to acquire, by an edge-side device, a first image set of each railway track fastener to be detected, the first image set including multiple first images of the railway track fastener to be detected with different exposure degrees. A second image processing module is configured to determine, by the edge-side device, a polarization correction coefficient of the first image set, and correct the first image set according to the polarization correction coefficient to obtain a second image of the railway track fastener to be detected, the information entropy and dynamic range of the second image being greater than those of the first image. A data uploading module is configured to determine a target edge-side device from multiple edge-side devices, receive, by the target edge-side device, the second image sent by other edge-side devices, and upload a second image set composed of multiple second images to a cloud server. A third image processing module is configured to acquire, by the cloud server, a light source correction coefficient and a vehicle body state correction coefficient of the second image set, and perform fusion and enhancement processing on the second image set according to the light source correction coefficient and the vehicle body state correction coefficient to obtain a target image of each railway track fastener to be detected, the definition and feature separability of the target image being greater than those of the second image. An identification module is configured to input the target image of each railway track fastener to be detected into a preset fastener defect detection model to obtain a defect detection result corresponding to each railway track fastener to be detected.

Images