Railway track fastener defect detection method and system based on image recognition

Abstract

The invention provides a railway track fastener defect detection method and system based on image recognition, and relates to the technical field of railway defect detection. The method comprises the following steps: acquiring a first to-be-identified image acquired by a first acquisition device in an acquisition period and a multi-source operation state parameter acquired by a second acquisition device in the acquisition period, correcting the first to-be-identified image according to the multi-source operation state parameter, and acquiring a second to-be-identified image, determining a time feature and a space feature of each second to-be-identified image, and determining a target identification image of each to-be-detected railway track fastener according to the time feature and the space feature of each second to-be-identified image; and inputting the target identification image of each to-be-detected railway track fastener into a preset fastener defect detection model to obtain a defect detection result corresponding to each to-be-detected railway track fastener. The invention aims to improve the problem that the existing method cannot improve the recognition accuracy of the fastener defect under the condition that the capacity of a fastener defect detection model is limited.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of railway defect detection, and in particular to a railway track fastener defect detection method and system based on image recognition. Background Art

[0002] Railway track fasteners are key connecting components in railway track structures, mainly used to firmly fix the rails to the sleepers, maintain the track gauge, bear the train load and transmit force to the sleepers and the roadbed. With the increase in train speed and load, the working environment of fasteners is becoming increasingly harsh, and the risk of fatigue damage, mechanical aging, external force impact and other problems is also increasing.

[0003] Traditional manual inspections are inefficient, experience-dependent and prone to omissions. Therefore, the use of technologies such as image recognition and machine learning to achieve efficient, accurate and real-time detection of fastener defects has become an important direction for the intelligent operation and maintenance of railway infrastructure. Summary of the invention

[0004] The embodiment of the present application provides a railway track fastener defect detection method and system based on image recognition. The embodiment of the present application adopts the following technical solutions: In a first aspect, an embodiment of the present application provides a railway track fastener defect detection method based on image recognition, the method comprising: acquiring a first image to be identified acquired by a first acquisition device in an acquisition period and a multi-source operating state parameter acquired by a second acquisition device in an acquisition period, the first acquisition device and the second acquisition device being arranged on a detection vehicle, and the multi-source operating state parameter being used to characterize the operating state of the detection vehicle; Correcting the first image to be identified according to the multi-source operating state parameters to obtain a second image to be identified; Determine the time characteristics and spatial characteristics of each second image to be identified, and determine the target identification image of each railway track fastener to be detected based on the time characteristics and spatial characteristics of each second image to be identified; The target recognition image of each railway track fastener to be inspected is input into a preset fastener defect detection model to obtain a defect detection result corresponding to each railway track fastener to be inspected.

[0005] In a possible implementation of the first aspect, obtaining a first image to be identified acquired by a first acquisition device in an acquisition period and a multi-source operating state parameter acquired by a second acquisition device in an acquisition period includes: Determine a first data acquisition frequency collected by the first acquisition device in an acquisition period according to a matching result between the running speed of the inspection vehicle and the image definition; Determining a second data acquisition frequency acquired by the second acquisition device in an acquisition period according to acquisition accuracy requirements of the operating state parameters and the degree of influence of the parameters on image stability; A first data acquisition frequency is used to acquire a first image to be identified, and a second data acquisition frequency is used to acquire multi-source operating status parameters.

[0006] In a possible implementation of the first aspect, the first image to be recognized is corrected according to the multi-source operating state parameter to obtain the second image to be recognized, including: Obtaining vehicle speed parameters, vehicle posture parameters and jitter information parameters from multi-source operating status parameters; Performing geometric distortion correction on the first image to be recognized according to the vehicle posture parameters to obtain a corrected image; According to the vehicle speed parameter and the jitter information parameter, time synchronization and motion blur correction are performed on the corrected image to obtain a second image to be recognized.

[0007] In a possible implementation of the first aspect, geometric distortion correction is performed on the first image to be recognized according to the vehicle posture parameter to obtain a corrected image, including: According to the vehicle posture parameters, construct a spatial posture matrix of the first acquisition device at the time of image acquisition; Determining a geometric correction model of the first image to be recognized according to the spatial posture matrix and the internal parameters of the first acquisition device; According to the geometric correction model, geometric correction is performed on the first image to be recognized to obtain a corrected image.

[0008] In a possible implementation of the first aspect, determining the temporal feature and the spatial feature of each second image to be recognized includes: Obtaining a shooting timestamp of each second image to be identified, and determining a time feature of the second image to be identified according to the shooting timestamp; The shooting coordinates of each second image to be recognized are obtained, and the spatial features of the second image to be recognized are determined according to the shooting coordinates.

[0009] In a possible implementation of the first aspect, determining a target recognition image of each railway track fastener to be detected according to the temporal features and spatial features of each second image to be recognized includes: Screening out a first candidate image set from a plurality of second images to be identified based on a matching result between the position information of the railway track fastener to be detected and the spatial features of the second image to be identified; According to the time feature of each candidate image in the first candidate image set, the candidate images are sorted according to the image acquisition time, and the second candidate image set is screened out from the first candidate image set in combination with the imaging angle, the illumination condition and the image clarity; Image registration and fusion are performed on all candidate images in the second candidate image set based on temporal features and spatial features to generate a target recognition image for each railway track fastener to be detected.

[0010] In a possible implementation of the first aspect, inputting the target recognition image of each railway track fastener to be inspected into a preset fastener defect detection model to obtain a defect detection result corresponding to each railway track fastener to be inspected includes: Inputting the target recognition image into the fastener defect detection model to obtain evaluation scores of different fastener defect types of the target recognition image; The fastener defect type with the highest evaluation score is determined as the fastener defect detection result of the railway track fastener to be inspected.

[0011] In a second aspect, the present application provides another railway track fastener defect detection system based on image recognition, the system comprising: An acquisition module, used to acquire a first image to be identified acquired by a first acquisition device in an acquisition period and a multi-source operating state parameter acquired by a second acquisition device in an acquisition period, wherein the first acquisition device and the second acquisition device are arranged on the detection vehicle, and the multi-source operating state parameter is used to characterize the operating state of the detection vehicle; A first image processing module, used for correcting the first image to be identified according to the multi-source operating state parameters to obtain a second image to be identified; A second image processing module, used to determine the time characteristics and spatial characteristics of each second image to be identified, and determine the target identification image of each railway track fastener to be detected according to the time characteristics and spatial characteristics of each second image to be identified; The defect detection module is used to input the target recognition image of each railway track fastener to be detected into a preset fastener defect detection model to obtain the defect detection result corresponding to each railway track fastener to be detected.

[0012] In a possible implementation of the second aspect, the acquisition module includes: A first frequency determination submodule, used to determine a first data acquisition frequency collected by the first acquisition device in an acquisition period according to a running speed of the detection vehicle and an image definition matching result; A second frequency determination submodule, used to determine the second data acquisition frequency acquired by the second acquisition device in an acquisition period according to the acquisition accuracy requirement of the operating state parameter and the influence degree on the image stability; The acquisition submodule is used to acquire the first image to be identified by adopting the first data acquisition frequency, and to acquire the multi-source operation status parameters by adopting the second data acquisition frequency.

[0013] In a possible implementation of the second aspect, the first image processing module includes: An acquisition submodule is used to acquire vehicle speed parameters, vehicle posture parameters and jitter information parameters from multi-source operating status parameters; A first image correction submodule is used to perform geometric distortion correction on the first image to be recognized according to the vehicle posture parameters to obtain a corrected image; The second image correction submodule is used to perform time synchronization and motion blur correction on the corrected image according to the vehicle speed parameter and the jitter information parameter to obtain a second image to be recognized.

[0014] In a third aspect, the present application also provides an electronic device, comprising: a memory and one or more processors, the memory being coupled to the processor; wherein the memory stores computer program code, the computer program code comprising computer instructions, and when the computer instructions are executed by the processor, the electronic device executes the method in any possible design mode of the first aspect mentioned above.

[0015] In a fourth aspect, the present application provides a computer-readable storage medium, comprising computer instructions; when the computer instructions are executed on an electronic device, the electronic device executes the method in the first aspect and any possible design thereof.

[0016] In a fifth aspect, the present application provides a computer program product. When the computer program product runs on an electronic device, the electronic device executes the method in the first aspect and any possible design thereof.

[0017] The present application provides a railway track fastener defect detection method based on image recognition. By acquiring the multi-source operating state parameters of the detection vehicle at the same time as image acquisition, and performing correction operations such as geometric distortion correction and deblurring processing on the image according to these parameters, it can effectively overcome the image blur, offset or distortion caused by the movement of the detection vehicle, and improve the quality of the original image data from the source. Secondly, by extracting the time characteristics and spatial characteristics of the second image to be identified and matching it with the spatial position of the track fastener to be detected, the corresponding relationship between the image and the fastener can be accurately determined, avoiding the misidentification caused by image drift or repeated coverage. At the same time, in view of the situation that a single image may be affected by illumination, imaging angle or occlusion, the present application further aligns and fuses multiple images with time and space differences but pointing to the same fastener, thereby generating a clearer and more stable target recognition image, providing high-quality input for subsequent recognition. This scheme effectively fills the problem of ignoring the impact of the data acquisition environment in the existing detection scheme by constructing a causal association mechanism between the dynamics of the detection vehicle and image interference, and breaks through the performance bottleneck faced by relying on image recognition model structure optimization or training data expansion. In addition, since the target recognition image is the fusion result after fully considering the vehicle state interference, its quality is more stable and the expression is more complete, which can significantly improve the robustness and recognition accuracy of the defect recognition model.

[0018] Among them, the technical effects of the second to fifth aspects refer to the technical effects of the first aspect and any of its embodiments, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 A schematic diagram of the steps of a railway track fastener defect detection method based on image recognition provided in an embodiment of the present application; Figure 2 A schematic diagram of the functional modules of a railway track fastener defect detection system based on image recognition provided in an embodiment of the present application. DETAILED DESCRIPTION

[0020] The terms used in the following embodiments are only for the purpose of describing specific embodiments, and are not intended to be used as limitations to the present application. As used in the specification and the appended claims of the present application, the singular expressions "one", "a kind of", "above", "the" and "this" are intended to also include, for example, "one or more" such expressions, unless there is an explicit contrary indication in the context. It should also be understood that in the following embodiments of the present application, "at least one", "one or more" refer to one or more (including two). The character " / " generally represents that the front and back associated objects are a kind of "or" relationship.

[0021] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments.

[0022] In the following, the terms "first", "second", etc. are used only for convenience of description and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first", "second", etc. may explicitly or implicitly include one or more of the features. In the description of this application, unless otherwise specified, "plurality" means two or more. For example, a plurality of processing units refers to two or more processing units.

[0023] In addition, in the embodiments of the present application, "upper", "lower", "left" and "right" are not limited to being defined relative to the orientation of the components schematically placed in the drawings. It should be understood that these directional terms can be relative concepts, which are used for description and clarification relative to the components, and can change accordingly according to the change in the orientation of the components placed in the drawings. In the drawings, for the sake of clarity, the thickness of the layers and regions is exaggerated, and the size ratio relationship between the parts in the drawings does not reflect the actual size ratio relationship.

[0024] In the embodiments of the present application, unless otherwise clearly specified and limited, the term "connection" should be understood in a broad sense. For example, "connection" can be a fixed connection, a detachable connection, or an integral connection; it can be a direct connection or an indirect connection through an intermediate medium. In addition, the term "electrical connection" can be a direct electrical connection or an indirect electrical connection through an intermediate medium.

[0025] In the embodiments of the present application, the term "module" is generally a functional structure divided according to logic, and the "module" can be implemented by pure hardware, or by a combination of software and hardware. In the embodiments of the present application, "and / or" describes the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, B exists alone, and A and B exist at the same time.

[0026] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0027] At present, when using models to detect defects in railway track fasteners, more emphasis is placed on optimizing the model structure and improving the training accuracy, such as using deeper neural networks, larger data sets, or more complex feature extraction algorithms to improve detection accuracy and generalization capabilities. However, with the development of this model-centric optimization method, the improvement of overall detection performance in actual scenarios has gradually become marginal, and it is unable to effectively handle recognition errors caused by unstable image acquisition conditions.

[0028] Specifically, when the image is blurred, the lighting changes, occlusion and perspective deviation often lead to insufficient quality of the original image, even a high-precision recognition model will find it difficult to stably output accurate results. Therefore, simply relying on the optimization of the model structure is difficult to meet the dual requirements of detection accuracy and stability in engineering practice.

[0029] In the current scheme, if you want to collect original images that meet the recognition accuracy requirements, you often rely on manual frame-by-frame screening or reshooting, which is not only time-consuming and labor-intensive, but also inefficient and easily affected by subjective judgment and resulting in omissions. Although some schemes use image acquisition equipment mounted on inspection vehicles for automatic acquisition, they generally ignore the significant impact of the inspection vehicle's own operating status on the image acquisition process.

[0030] In the actual operation process, the inspection vehicle may experience speed fluctuations, severe vibrations, uneven tracks, etc., which can easily lead to image blur, distortion, offset, changes in viewing angle, and even missing key parts. In addition, changes in the vehicle's posture (such as pitch and roll) will affect the shooting angle, causing misalignment between the image and the fastener position; changes in ambient light over time and position may also cause uneven image brightness, local overexposure, or shadow obstruction. The combined effect of these factors will make the quality of the collected images unstable, seriously affecting the input effect and recognition accuracy of the subsequent recognition model. Therefore, the image acquisition strategy that ignores the running status of the inspection vehicle is difficult to ensure the availability and consistency of image data from the source, which has become a key bottleneck restricting the stable operation of the fastener defect automatic recognition system.

[0031] For example, during the original image acquisition process, the track fastener A to be inspected is in a normal state. However, due to factors such as speed changes, vibration shocks, or posture deviations during the operation of the inspection vehicle itself, the key parts of the fastener A in the acquired image are missing or the overall image is blurred. At this point, even if the defect recognition model used later has extremely high recognition accuracy, it is difficult to make accurate judgments under the premise of incomplete input data or distorted information. It is very easy to misjudge the normal fastener as an abnormal state, thereby affecting the accuracy and credibility of the entire inspection system.

[0032] Based on this, the inventive concept of the present application is proposed: while collecting the original images of railway track fasteners, the operating status parameters of the inspection vehicle are obtained, and by modeling and analyzing the operating status corresponding to the image acquisition time point, the blur, offset, distortion and other problems caused by the vehicle body movement during the image acquisition process are dynamically corrected, thereby effectively improving the accuracy and reliability of railway track fastener defect detection.

[0033] Reference Figure 1 The embodiment of the present invention provides a railway track fastener defect detection method based on image recognition, which is applied to a host computer and may specifically include the following steps: S101: Acquire a first image to be recognized acquired by a first acquisition device in an acquisition period and a multi-source operating state parameter acquired by a second acquisition device in an acquisition period.

[0034] In this embodiment, the first acquisition device and the second acquisition device are arranged on the inspection vehicle. The first acquisition device can be an image acquisition device such as a high-resolution industrial camera, a linear array camera or a multi-camera array, which is used to continuously capture images of the railway tracks and fasteners along the line during the movement of the inspection vehicle, and obtain the original image data containing the track fasteners to be inspected. The second acquisition device can include an inertial measurement unit, a GPS positioning module, an acceleration sensor, an angular velocity sensor, a posture sensor, a vibration sensor and other operating status acquisition modules, which are used to obtain the speed, acceleration, position, posture angle, vibration intensity and other multi-source operating status parameters of the inspection vehicle in real time during the image acquisition cycle, so as to fully characterize the dynamic operating state of the inspection vehicle.

[0035] Specific steps may include: S1011: determining a first data acquisition frequency collected by a first acquisition device in an acquisition period according to a matching result between a running speed of the inspection vehicle and image definition; S1012: Determine a second data acquisition frequency acquired by a second acquisition device in an acquisition period according to acquisition accuracy requirements of the operating state parameters and the degree of influence of the parameters on image stability; S1013: Acquire a first image to be recognized using a first data acquisition frequency, and acquire a multi-source operating status parameter using a second data acquisition frequency.

[0036] In the implementation methods of S1011 to S1013, when determining the first data acquisition frequency, the running speed information of the inspection vehicle during the acquisition period is first obtained, and combined with the minimum requirements of the image recognition model for image clarity (including image resolution, target area integrity, edge sharpness and other indicators), a matching model and corresponding relationship between speed and clarity can be constructed. It is used to evaluate the impact of image acquisition frequency on image quality at different vehicle speeds, ensure that each frame of the image contains at least one complete railway track fastener target, and avoid image smearing caused by excessive vehicle speed or target loss caused by excessive sampling intervals. Through the matching results, the optimal first data acquisition frequency is dynamically determined, so that the image acquisition process can meet the recognition accuracy requirements without wasting resources. It should be noted that the number of inspection vehicles can be multiple, and multiple inspection vehicles can perform data acquisition tasks at different times.

[0037] When determining the second data acquisition frequency, analyze the impact intensity and change frequency of each operating state parameter on the stability of image acquisition during the movement of the inspection vehicle, evaluate its change speed on the time scale and its potential interference with image quality, and combine the image acquisition frequency and image blur tolerance requirements to determine the minimum sampling time interval of the operating state parameter, so as to set the sampling frequency of the second acquisition device, so that it can accurately capture any dynamic interference factors that may cause image distortion, blur or offset during the image acquisition process, and ensure that the collected state data has sufficient timeliness and resolution. It should be noted that during the data acquisition process, the second data acquisition frequency can be dynamically adjusted according to the changes in the operating conditions of the inspection vehicle.

[0038] For example, when the inspection vehicle passes through a turnout, a curved section, or an area with uneven tracks, the acceleration and attitude angle change frequently and with a large amplitude, which can easily cause image blur, offset, or even partial occlusion. If a static fixed sampling frequency is still used, it may not be possible to capture key disturbance information in time, affecting the accuracy of subsequent image correction. By dynamically adjusting the second data acquisition frequency, the sampling frequency is automatically increased when the vehicle dynamics change dramatically, and the frequency is appropriately reduced when the state is stable, which can not only achieve high-precision tracking of interference factors, but also reduce system resource consumption.

[0039] After completing the frequency configuration of the first acquisition device and the second acquisition device, the system enters the actual data acquisition stage. The image acquisition device takes images of the rail fastener area according to the set first data acquisition frequency to obtain the original image sequence containing the target to be detected. At the same time, the running status acquisition device synchronously and continuously records the dynamic running status parameters of the inspection vehicle at the second data acquisition frequency. The two types of data can be accurately aligned through the timestamp mechanism, providing high-quality time series basic data support for subsequent image quality analysis, spatiotemporal feature modeling and defect recognition algorithms.

[0040] S102: Correcting the first image to be recognized according to the multi-source operating state parameters to obtain a second image to be recognized.

[0041] In this embodiment, by analyzing the multi-source operating status parameters of the detection vehicle during the image acquisition process, a causal correlation model between image distortion or blur and vehicle dynamics is established, and then the original image affected by interference is geometrically corrected, deblurred or quality screened to improve the image quality and obtain a more stable second image to be identified that can be used for subsequent identification and analysis.

[0042] Most existing solutions focus on the structural optimization and training set expansion of image recognition models, but ignore that the quality of original data is the basic premise of model performance. This application controls the image quality from the source by introducing the running status information of the inspection vehicle into the image processing process. On the one hand, it can make accurate corrections in the case of severe image distortion to avoid the loss of effective samples. On the other hand, it also provides a clearer, standardized, and unified image input in time and space for subsequent target recognition, significantly enhancing the robustness and practicality of the overall recognition system, thus breaking through the existing bottleneck of accuracy improvement brought by relying solely on model architecture optimization.

[0043] The specific steps may include: S1021: Acquire vehicle speed parameters, vehicle posture parameters, and jitter information parameters from multi-source operating state parameters; S1022: performing geometric distortion correction on the first image to be recognized according to the vehicle posture parameter to obtain a corrected image; S1023: Perform time synchronization and motion blur correction on the corrected image according to the vehicle speed parameter and the jitter information parameter to obtain a second image to be recognized.

[0044] In the implementation of S1021 to S1023, the vehicle speed parameter may include the instantaneous speed, average speed or acceleration of the detection vehicle during the acquisition period, etc., which is used to reflect the movement amplitude of the vehicle during image acquisition. The vehicle attitude parameters may include attitude angle information such as pitch angle, roll angle, yaw angle, etc., which are used to describe the attitude changes of the detection vehicle in space, and have a direct impact on the tilt and geometric distortion of the image. The jitter information parameters may include short-term disturbance information such as high-frequency vibration and structural impact, which can be used to measure the dynamic interference source of image blur. During the operation of the detection vehicle, due to factors such as track undulations, curves passing or uneven equipment installation, the image may be tilted, distorted, skewed and other geometric deformation problems at the moment of shooting. Therefore, a geometric mapping relationship between the three-dimensional attitude of the detection vehicle and the image plane can be constructed to geometrically restore the original image to obtain a corrected image.

[0045] When the inspection vehicle is in rapid motion, the captured images will have obvious linear smear and local jitter blur, especially when the acquisition frequency is insufficient or the exposure time is long. To this end, based on the vehicle speed and direction information, the blur kernel caused by linear motion of each pixel in the image is estimated, and blur compensation is performed using deconvolution, deep deblurring network or image restoration algorithm. The slight blur caused by vibration (such as slight ghosting, edge shaking) is filtered in the frequency domain or multiple frames are fused for clarity enhancement. The final second image to be identified has high geometric fidelity and edge clarity, which can meet the quality input requirements of the image recognition model and improve the accuracy of defect recognition.

[0046] In a feasible implementation manner, the specific steps of performing geometric distortion correction on the first image to be recognized according to the vehicle posture parameter and obtaining the corrected image may include: According to the vehicle posture parameters, construct a spatial posture matrix of the first acquisition device at the time of image acquisition; Determining a geometric correction model of the first image to be recognized according to the spatial posture matrix and the internal parameters of the first acquisition device; According to the geometric correction model, geometric correction is performed on the first image to be recognized to obtain a corrected image.

[0047] In this embodiment, first, the spatial posture information of the detection vehicle at the time of image acquisition is collected by the posture sensor, and then the spatial posture matrix of the first acquisition device at the moment of shooting is constructed to describe its rotation and position state in three-dimensional space; then, in combination with the spatial posture matrix and the imaging optical characteristics and structural configuration of the first acquisition device itself, a transformation model for image geometric correction is established, and the transformation model can describe the angle offset, perspective distortion, projection error and other problems caused by the change of the device posture in the image; finally, according to the transformation model, each pixel point in the original image is geometrically demapped and the image is resampled, and the distorted image is restored to the image perspective under the standard posture, so as to obtain a corrected image with real structure, clear edges and accurate geometric relationship.

[0048] S103: Determine the time characteristics and space characteristics of each second image to be identified, and determine the target identification image of each railway track fastener to be detected according to the time characteristics and space characteristics of each second image to be identified.

[0049] In this embodiment, during the continuous operation of the inspection vehicle, the same railway track fastener to be inspected will be photographed multiple times by the first acquisition device at different times or collected by different first acquisition devices at short time intervals, thereby generating multiple image data with different acquisition times and slightly different viewing angles. Due to the influence of factors such as the running state, posture changes, vibration interference, and lighting conditions of the inspection vehicle, these images may still have a certain degree of blur, offset, or local information loss even after state parameter correction. In order to ensure that the image ultimately used for defect identification has higher integrity and clarity, it is necessary to align the multiple second images to be identified obtained by taking the same fastener at different times in time and space, fuse the content, and optimize them to construct a target recognition image that is closer to the ideal imaging effect. The main difference between the target recognition image and the second image to be identified is that it is unique and targeted, can establish a one-to-one mapping relationship with a specific track fastener, and is the direct input data for the subsequent defect recognition model to perform detection and judgment. The specific steps for determining the target recognition image may include: S1031: selecting a first candidate image set from a plurality of second images to be identified based on a matching result between the position information of the railway track fastener to be detected and the spatial features of the second image to be identified; S1032: sorting the candidate images in the first candidate image set according to the time feature of each candidate image in the first candidate image set according to the image acquisition time, and combining the imaging angle, the illumination condition and the image definition, to select a second candidate image set from the first candidate image set; S1033: Perform image registration and fusion on all candidate images in the second candidate image set based on temporal features and spatial features to generate a target recognition image for each railway track fastener to be detected.

[0050] In the implementation methods of S1031 to S1033, before determining the target identification image of each railway track fastener to be detected, it is first necessary to perform feature calibration on all second images to be identified in the time and space dimensions to achieve accurate mapping between images and physical fasteners. Specifically, during the acquisition process of each second image to be identified, a corresponding shooting timestamp will be generated, which records the exact time point when the image was acquired. By analyzing the relationship between the image acquisition time and the motion trajectory of the inspection vehicle, the temporal distribution of the fasteners contained in the image can be inferred, and then its time characteristics can be determined. At the same time, the geographic or track position coordinate information corresponding to the image will also be recorded during image acquisition. The shooting coordinates can be obtained through the positioning system and combined with the structural data of the track line to restore the spatial distribution characteristics of the fasteners in the image.

[0051] After obtaining the temporal and spatial features of each second image to be identified, firstly, the spatial features of each second image to be identified are matched according to the known position information of the railway track fastener to be detected. The spatial feature is usually the shooting coordinates of the image. By calculating the spatial distance and relative orientation relationship between the image shooting point and the target fastener, the images covering the target fastener or its adjacent area at the shooting position are screened out. These images constitute the first candidate image set. This step ensures that the subsequent processing is only focused on images that may contain the target fastener, and the data of irrelevant areas are eliminated, which effectively improves the processing efficiency.

[0052] Based on the first candidate image set, the system further combines the time features (shooting timestamp) of each image to sort the images in chronological order, and comprehensively considers multiple quality dimensions such as imaging angle (such as image shooting posture), lighting conditions (such as whether there is strong reflection or shadow), image clarity (such as whether there is motion blur, out-of-focus distortion), etc., to optimize the images. The screening result is the second candidate image set, which represents a subset of images that are representative in terms of time coverage and have high availability in quality evaluation.

[0053] Based on the images in the second candidate image set, an image registration operation is performed. Registration refers to aligning images taken at different times and angles through geometric transformation so that they can be accurately superimposed in the same coordinate reference system in space. During the registration process, time features are used to manage the image sequence in time, and spatial features are used to guide coordinate transformation to ensure the consistency of image content. Subsequently, the information of multiple images is integrated into a target recognition image with higher resolution, more complete structure, and lower noise through multi-image fusion technology (such as weighted averaging, maximum information retention fusion, image completion, etc.). This image can more realistically and accurately reflect the appearance and potential defects of the target fastener, and is the key basic data for subsequent defect recognition model analysis.

[0054] As an example, taking a railway track fastener A to be detected as an example, how to filter and fuse the second image to be identified by constructing the first candidate image set and the second candidate image set in the process of generating the target recognition image is explained in detail. Track fastener A is located at a fixed track position (for example, GPS coordinates Xa, Ya). The position information of fastener A is matched with the spatial features of all second images to be identified. By judging whether the image field of view contains the Xa, Ya position, all second images to be identified whose spatial positions overlap or are close to fastener A are filtered out. These second images to be identified constitute the first candidate image set of fastener A. Assume that a total of 5 images are filtered out, which are taken at five different time points: T1, T2, T3, T4, and T5.

[0055] Then, the system further evaluates the quality of these five images. The evaluation content includes: time feature sorting: sorting the images by shooting time T1~T5 to construct a time series; imaging angle analysis: analyzing the posture of the inspection vehicle when the image is shot (such as tilt, pitch), and selecting images with a more correct perspective; lighting condition screening: eliminating images with underexposure or overexposure and strong shadows; image clarity evaluation: using clarity evaluation functions (such as Laplace variance, edge clarity) to evaluate image quality and filter out images with obvious blur. Assume that images T2, T3, and T4 have the best quality and have better imaging angles and clarity, which constitute the second candidate image set of fastener A.

[0056] Finally, the three images T2, T3, and T4 are registered: their spatial features are used to geometrically align them in a unified reference system to ensure that the position of fastener A in each image completely overlaps. Based on the registration, the information of the three images is integrated through fusion algorithms (such as multiple exposure fusion and pixel-level weighted average) to compensate for the illumination fluctuations, local occlusions, or information missing problems that may exist in a single image, and finally a high-quality, high-definition, and detailed target recognition image is generated for subsequent defect detection and analysis.

[0057] S104: Inputting the target recognition image of each railway track fastener to be inspected into a preset fastener defect detection model to obtain a defect detection result corresponding to each railway track fastener to be inspected.

[0058] The defect detection results may include normal, rotated, reversed, offset, damaged, missing and other different results. The defect detection results corresponding to each railway track fastener to be detected are obtained by inputting the target recognition image of each railway track fastener to be detected into the preset fastener defect detection model. The specific implementation steps may include: S1041: inputting the target recognition image into a fastener defect detection model to obtain evaluation scores of different fastener defect types of the target recognition image; S1042: Determine the fastener defect type with the highest evaluation score as the fastener defect detection result of the railway track fastener to be detected.

[0059] In the implementation of S1041 to S1042, the target recognition image corresponding to each railway track fastener to be inspected is input into a preset fastener defect detection model. The model will identify and score the possible defect types in the image and output the evaluation scores of each type of defects.

[0060] As an example, for a target recognition image of a fastener, the model may output the following evaluation results: normal 0.08, rotation 0.15, offset 0.10, damage 0.60, missing 0.07. The evaluation score reflects the model's confidence in the different defect types of the image. The higher the value, the more likely the model believes that the image belongs to this type of defect. In this example, the highest score for the "damage" type is 0.60, so "damage" can be determined as the defect detection result of the fastener to be detected.

[0061] The railway track fastener defect detection method based on image recognition provided by the present application can effectively overcome the image blur, offset or distortion caused by the motion of the detection vehicle by acquiring the multi-source operating state parameters of the detection vehicle at the same time as the image acquisition, and perform correction operations such as geometric distortion correction and deblurring processing on the image according to these parameters, so as to improve the quality of the original image data from the source. Secondly, by extracting the time characteristics and spatial characteristics of the second image to be identified and matching it with the spatial position of the track fastener to be detected, the corresponding relationship between the image and the fastener can be accurately determined, avoiding the misidentification caused by image drift or repeated coverage. At the same time, in view of the situation that a single image may be affected by illumination, imaging angle or occlusion, the present application further aligns and fuses multiple images with time and space differences but pointing to the same fastener, thereby generating a clearer and more stable target recognition image, providing high-quality input for subsequent recognition. This scheme effectively fills the problem of ignoring the impact of the data acquisition environment in the existing detection scheme by constructing a causal association mechanism between the dynamics of the detection vehicle and image interference, and breaks through the performance bottleneck faced by relying on the optimization of the image recognition model structure or the expansion of training data. In addition, since the target recognition image is the fusion result after fully considering the vehicle state interference, its quality is more stable and the expression is more complete, which can significantly improve the robustness and recognition accuracy of the defect recognition model.

[0062] In the second aspect, based on the same inventive concept, refer to Figure 2 , shows a railway track fastener defect detection system 200 based on image recognition provided by an embodiment of the present application, the system comprising: An acquisition module 201 is used to acquire a first image to be identified acquired by a first acquisition device in an acquisition period and a multi-source operating state parameter acquired by a second acquisition device in an acquisition period, wherein the first acquisition device and the second acquisition device are arranged on a detection vehicle, and the multi-source operating state parameter is used to characterize the operating state of the detection vehicle; The first image processing module 202 is used to correct the first image to be identified according to the multi-source operating state parameters to obtain a second image to be identified; The second image processing module 203 is used to determine the time characteristics and spatial characteristics of each second image to be identified, and determine the target identification image of each railway track fastener to be detected according to the time characteristics and spatial characteristics of each second image to be identified; The defect detection module 204 is used to input the target recognition image of each railway track fastener to be detected into a preset fastener defect detection model to obtain a defect detection result corresponding to each railway track fastener to be detected.

[0063] In a possible implementation of the second aspect, the acquisition module includes: A first frequency determination submodule, used to determine a first data acquisition frequency collected by the first acquisition device in an acquisition period according to a running speed of the detection vehicle and an image definition matching result; A second frequency determination submodule, used to determine the second data acquisition frequency acquired by the second acquisition device in an acquisition period according to the acquisition accuracy requirement of the operating state parameter and the influence degree on the image stability; The acquisition submodule is used to acquire the first image to be identified by adopting the first data acquisition frequency, and to acquire the multi-source operation status parameters by adopting the second data acquisition frequency.

[0064] In a possible implementation of the second aspect, the first image processing module includes: An acquisition submodule is used to acquire vehicle speed parameters, vehicle posture parameters and jitter information parameters from multi-source operating status parameters; A first image correction submodule is used to perform geometric distortion correction on the first image to be recognized according to the vehicle posture parameters to obtain a corrected image; The second image correction submodule is used to perform time synchronization and motion blur correction on the corrected image according to the vehicle speed parameters and the jitter information parameters, so as to obtain the second image to be identified. It should be noted that the specific implementation manner of the railway track fastener defect detection system 200 based on image recognition in the embodiment of the present application refers to the specific implementation manner of the railway track fastener defect detection method based on image recognition proposed in the first aspect of the embodiment of the present application, and will not be repeated here.

[0065] The present application also provides an electronic device, which may include: a memory and one or more processors. The memory and the processor are coupled. The memory is used to store computer program code, and the computer program code includes computer instructions. When the processor executes the computer instructions, the electronic device can perform each function or step in the above method embodiment.

[0066] This embodiment also provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed on an electronic device, the electronic device executes each function or step in the above method embodiment.

[0067] This embodiment also provides a computer program product. When the computer program product is run on a computer, it enables the computer to execute each function or step in the above method embodiment.

[0068] Among them, the electronic device, computer-readable storage medium, and computer program product provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0069] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When a computer instruction or computer program is loaded or executed on a computer, a process or function according to an embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable system. The computer program or instruction can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer program or instruction can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0070] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0071] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0072] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems, systems and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0073] In the several embodiments provided in the present application, it should be understood that the disclosed systems, systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the system or unit can be electrical, mechanical or other forms.

[0074] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0075] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0076] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0077] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A railway track fastener defect detection method based on image recognition, characterized in that: The method comprises: Acquire a first image to be identified acquired by a first acquisition device in an acquisition period and a multi-source operating state parameter acquired by a second acquisition device in the acquisition period, wherein the first acquisition device and the second acquisition device are arranged on a detection vehicle, and the multi-source operating state parameter is used to characterize the operating state of the detection vehicle; According to the multi-source operating state parameters, the first image to be identified is corrected to obtain a second image to be identified; Determine the time characteristics and spatial characteristics of each of the second images to be identified, and determine the target identification image of each railway track fastener to be detected based on the time characteristics and spatial characteristics of each of the second images to be identified; The target recognition image of each railway track fastener to be inspected is input into a preset fastener defect detection model to obtain a defect detection result corresponding to each railway track fastener to be inspected.

2. The railway track fastener defect detection method based on image recognition according to claim 1 is characterized in that: The step of acquiring the first image to be identified acquired by the first acquisition device in an acquisition period and the multi-source operating status parameters acquired by the second acquisition device in the acquisition period includes: Determining a first data acquisition frequency acquired by the first acquisition device in the acquisition period according to a matching result between the running speed of the detection vehicle and the image clarity; Determining a second data acquisition frequency acquired by the second acquisition device in the acquisition period according to the acquisition accuracy requirement of the multi-source operating state parameters and the degree of influence of the parameters on image stability; The first data acquisition frequency is used to acquire the first image to be identified, and the second data acquisition frequency is used to acquire the multi-source operating status parameter.

3. The railway track fastener defect detection method based on image recognition according to claim 1, characterized in that: The step of correcting the first image to be identified according to the multi-source operating state parameter to obtain a second image to be identified includes: Acquire a vehicle speed parameter, a vehicle posture parameter, and a jitter information parameter from the multi-source operating state parameters; Performing geometric distortion correction on the first image to be recognized according to the vehicle posture parameter to obtain a corrected image; According to the vehicle speed parameter and the jitter information parameter, time synchronization and motion blur correction are performed on the corrected image to obtain the second image to be recognized.

4. The railway track fastener defect detection method based on image recognition according to claim 3 is characterized in that: The step of performing geometric distortion correction on the first image to be recognized according to the vehicle posture parameter to obtain a corrected image includes: Constructing a spatial posture matrix of the first acquisition device at the time of image acquisition according to the vehicle posture parameters; Determining a geometric correction model of the first image to be recognized according to the spatial posture matrix and the internal parameters of the first acquisition device; According to the geometric correction model, geometric correction is performed on the first image to be recognized to obtain the corrected image.

5. The railway track fastener defect detection method based on image recognition according to claim 1, characterized in that: The determining of the temporal features and spatial features of each of the second images to be recognized comprises: Acquire a shooting timestamp of each of the second images to be identified, and determine a time feature of the second images to be identified according to the shooting timestamp; The shooting coordinates of each of the second images to be recognized are obtained, and the spatial features of the second images to be recognized are determined according to the shooting coordinates.

6. The railway track fastener defect detection method based on image recognition according to claim 1, characterized in that: Determining the target recognition image of each railway track fastener to be detected according to the temporal features and spatial features of each second image to be recognized includes: Screening out a first candidate image set from a plurality of second images to be identified according to a matching result between the position information of the railway track fastener to be detected and the spatial features of the second image to be identified; According to the time feature of each candidate image in the first candidate image set, the candidate images are sorted according to the image acquisition time, and combined with the imaging angle, the illumination condition and the image clarity, a second candidate image set is screened out from the first candidate image set; Image registration and fusion are performed on all candidate images in the second candidate image set based on temporal features and spatial features to generate a target recognition image for each railway track fastener to be detected.

7. The railway track fastener defect detection method based on image recognition according to claim 1, characterized in that: The step of inputting the target recognition image of each railway track fastener to be inspected into a preset fastener defect detection model to obtain a defect detection result corresponding to each railway track fastener to be inspected includes: Inputting the target recognition image into the fastener defect detection model to obtain evaluation scores of different fastener defect types of the target recognition image; The fastener defect type with the highest evaluation score is determined as the fastener defect detection result of the railway track fastener to be detected.

8. A railway track fastener defect detection system based on image recognition, used to implement the method of any one of claims 1 to 7, characterized in that: The system comprises: An acquisition module, used to acquire a first image to be identified acquired by a first acquisition device in an acquisition period and a multi-source operating state parameter acquired by a second acquisition device in the acquisition period, wherein the first acquisition device and the second acquisition device are arranged on a detection vehicle, and the multi-source operating state parameter is used to characterize the operating state of the detection vehicle; A first image processing module, used for correcting the first image to be identified according to the multi-source operating state parameters to obtain a second image to be identified; A second image processing module, used to determine the time characteristics and spatial characteristics of each of the second images to be identified, and determine a target identification image of each railway track fastener to be detected based on the time characteristics and spatial characteristics of each of the second images to be identified; The defect detection module is used to input the target recognition image of each railway track fastener to be detected into a preset fastener defect detection model to obtain the defect detection result corresponding to each railway track fastener to be detected.

9. The railway track fastener defect detection system based on image recognition according to claim 8, characterized in that: The acquisition module comprises: A first frequency determination submodule, used to determine a first data acquisition frequency acquired by the first acquisition device in the acquisition period according to a matching result between the running speed of the detection vehicle and the image definition; A second frequency determination submodule, configured to determine a second data acquisition frequency acquired by the second acquisition device in the acquisition period according to the acquisition accuracy requirement of the operating state parameter and the degree of influence of the operating state parameter on the image stability; The acquisition submodule is used to acquire the first image to be identified by adopting the first data acquisition frequency, and to acquire the multi-source operation status parameter by adopting the second data acquisition frequency.

10. The railway track fastener defect detection system based on image recognition according to claim 8, characterized in that: The first image processing module comprises: An acquisition submodule, used to acquire a vehicle speed parameter, a vehicle posture parameter and a jitter information parameter from the multi-source operating state parameters; A first image correction submodule, used for performing geometric distortion correction on the first image to be recognized according to the vehicle posture parameter to obtain a corrected image; The second image correction submodule is used to perform time synchronization and motion blur correction on the corrected image according to the vehicle speed parameter and the jitter information parameter to obtain the second image to be recognized.

Images