Railway infrastructure defect detection method and system based on multi-view vision

Abstract

The application provides a railway infrastructure defect detection method and system based on multi-view vision, relates to the technical field of data processing, and comprises the following steps: acquiring basic information of a railway inspection vehicle, arranging sensors in combination with a target inspection line, obtaining a completed visual sensor array, obtaining an infrastructure surface image set, acquiring a historical infrastructure surface defect image set and a historical infrastructure surface defect identification result set to construct a surface defect identification prototype, obtaining a surface defect identification prototype set, matching and identifying the infrastructure surface image set by using the surface defect identification prototype set, and obtaining a surface defect identification result. The application solves the technical problem that the existing technology lacks multi-dimensional comprehensive analysis for railway infrastructure defect detection, resulting in low reliability of the detection result, and achieves the technical effect of improving defect detection accuracy and reliability.

Description

Technical Field

[0001] This application relates to the field of data processing technology, specifically to a method and system for detecting defects in railway infrastructure based on multi-view vision. Background Technology

[0002] Existing railway inspection technologies commonly employ methods including manual inspection, mechanized inspection, and detection using single sensors such as lidar or vision sensors. These methods largely rely on manual or partial data collection, presenting significant limitations. Manual inspection requires substantial manpower, is slow, and prone to oversights. While mechanized inspection improves efficiency, it struggles to meet the demands for high precision and comprehensive monitoring. Furthermore, the application of single sensors typically fails to fully cover the health status of railway infrastructure, resulting in the inability to promptly and accurately identify potential problems. Summary of the Invention

[0003] This application provides a method and system for detecting defects in railway infrastructure based on multi-view vision, which solves the technical problem that the lack of multi-dimensional comprehensive analysis in the existing technology for defect detection of railway infrastructure leads to low reliability of detection results.

[0004] In view of the above problems, on the one hand, this application provides a method for detecting defects in railway infrastructure based on multi-view vision, the method comprising: The process involves acquiring basic information about railway inspection vehicles, deploying sensors along the target inspection line, and obtaining a completed visual sensor array. The visual sensor array is then used to collect surface images of the infrastructure along the target inspection line, resulting in a set of infrastructure surface images. A set of historical infrastructure surface defect images and a set of historical infrastructure surface defect identification results are then acquired to construct a surface defect identification prototype set. Finally, the surface defect identification prototype set is used to match and identify the infrastructure surface images, yielding surface defect identification results.

[0005] On the other hand, this application also provides a railway infrastructure defect detection system based on multi-view vision, the system comprising: The system includes a visual sensor array acquisition module for acquiring basic information about railway inspection vehicles, deploying sensors along the target inspection line, and obtaining a completed visual sensor array; an infrastructure surface image set acquisition module for using the visual sensor array to collect surface images of the infrastructure along the target inspection line, and obtaining an infrastructure surface image set; a recognition prototype construction module for acquiring a set of historical infrastructure surface defect images and a set of historical infrastructure surface defect recognition results to construct a surface defect recognition prototype set; and a surface defect recognition result acquisition module for using the surface defect recognition prototype set to match and recognize the infrastructure surface image set, and obtaining the surface defect recognition result.

[0006] One or more technical solutions provided in this application have at least the following beneficial effects: This application acquires basic information about railway inspection vehicles, deploys sensors along the target inspection line to obtain a completed visual sensor array, and then uses this array to collect surface images of the infrastructure along the target inspection line, resulting in a set of infrastructure surface images. Furthermore, it acquires a set of historical infrastructure surface defect images and a set of historical infrastructure surface defect identification results to construct a surface defect identification prototype set. Finally, it uses this prototype set to match and identify the infrastructure surface images, obtaining the surface defect identification results. This achieves the technical effect of improving the reliability and accuracy of defect detection.

[0007] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0008] Figure 1 This is a flowchart illustrating the multi-view vision-based defect detection method for railway infrastructure provided in this application embodiment.

[0009] Figure 2 A schematic diagram of the structure of a railway infrastructure defect detection system based on multi-view vision provided in an embodiment of this application.

[0010] Figure labeling: Visual sensor array acquisition module 11, infrastructure surface image set acquisition module 12, identification prototype construction module 13, surface defect identification result acquisition module 14. Detailed Implementation

[0011] This application provides a method and system for detecting defects in railway infrastructure based on multi-view vision, which solves the technical problem in the prior art that the lack of multi-dimensional comprehensive analysis in defect detection of railway infrastructure leads to low reliability of detection results.

[0012] Example 1, as Figure 1 As shown in the figure, this application provides a method for detecting defects in railway infrastructure based on multi-view vision. The method includes: Step S100: Obtain basic information of railway inspection vehicles, deploy sensors in conjunction with the target inspection line, and obtain the completed visual sensor array. Furthermore, by acquiring basic information about the railway inspection vehicle and deploying sensors along the target inspection line to obtain a completed visual sensor array, step S100 in this embodiment of the application also includes: The basic information is extracted according to a preset set of basic parameters to obtain the set of basic parameters; The basic information is extracted according to the preset deployment configuration indicators to obtain the number of sensor interface locations and power supply capacity; The target inspection route is analyzed according to the preset route indicators to obtain the inspection route indicators. The visual sensor array is obtained by deploying sensors based on the number of sensor interface locations and power supply capacity, combined with the basic parameter set and the inspection line indicators.

[0013] Furthermore, the preset set of basic parameters includes vehicle type, number of carriages, vehicle length, vehicle height, and vehicle width.

[0014] In the embodiments of this application, the basic information refers to the technical specifications and operating parameters of the railway inspection vehicle, including basic data such as vehicle type, number of carriages, and vehicle dimensions. This basic information is crucial for determining how to install sensors and deploy lidar equipment. The railway inspection vehicle can be any vehicle required for railway inspection. The target inspection line refers to a predetermined railway line, along which the railway inspection vehicle will conduct its inspection.

[0015] Preferably, the sensor deployment is carried out by considering the track condition of the target inspection route and the actual situation of the railway inspection vehicle, resulting in the completed visual sensor array. The visual sensor array consists of multiple cameras or image sensors and is typically positioned at different locations on the railway inspection vehicle (e.g., front, rear, sides) to cover a wider detection area.

[0016] In one possible embodiment, the preset basic parameter set is a set of parameters pre-defined by those skilled in the art to describe various basic characteristics of the railway inspection vehicle, including vehicle type, number of carriages, vehicle length, vehicle height, and vehicle width. The preset deployment configuration indicators are indicators pre-defined by those skilled in the art to describe the equipment deployment of the railway inspection vehicle, including the number of interface locations and power supply capacity. The preset route indicators are indicators pre-defined by those skilled in the art to describe the target inspection route, including route length, gradient, curve radius, etc.

[0017] Preferably, the basic information is extracted according to the preset basic parameters and the preset deployment configuration indicators to obtain the set of basic parameters, the number of sensor interface locations, and the power supply capacity. Here, the power supply capacity refers to the maximum power of the power system on the inspection vehicle that can support the sensors.

[0018] For example, suppose there is a railway inspection vehicle, a diesel locomotive, 20 meters long, 3 meters wide, and 4 meters high, with 5 carriages. This vehicle will inspect a railway line containing straight sections, curves, and tunnels. Assume the vehicle is equipped with a visual sensor array (6 high-definition cameras). Furthermore, the vehicle's power supply capacity is 1000 watts (W), while each camera requires approximately 50W of power. The number of sensor interfaces is limited to 8. The target inspection line's metrics are obtained: line length: 50 kilometers; gradient: some areas have a gradient of 3%; curve radius: some areas have a curve radius of 200 meters, others 500 meters; special locations: 3 tunnels require focused monitoring. Based on the basic information and the inspection line's metrics, sensor deployment begins.

[0019] The vehicle has eight sensor interfaces, and each camera requires power. We need to ensure that the total power consumption of these devices does not exceed 1000W. Six vision sensors (50W each) consume a total of 300W. This is far below the vehicle's 1000W power supply capacity, therefore, the power supply is sufficient.

[0020] Based on the dimensions of the train car (20 meters long and 3 meters wide), the installation location of each vision sensor can be reasonably determined. Since there are five cars, the sensors can be installed in appropriate positions at the front, rear, and sides to ensure data on the railway infrastructure is acquired from multiple angles. Because the car body is 3 meters wide, the sensors can be slightly offset to avoid blind spots. The sensor deployment should be optimized according to the characteristics of different sections of the track, such as gradient and curve radius.

[0021] Straight Roads: On straight road sections up to 50 kilometers long, vehicles can acquire images of infrastructure using vision sensors at the front and rear. Curved Roads: On curves with a radius of 200 meters, additional vision sensors need to be deployed on both sides of the vehicle to ensure that images of infrastructure on both sides can be captured. In this case, two vision sensors may be mounted on the side windows, facing left and right respectively. Steep Slopes. Special Locations (Tunnels): At the locations of three tunnels, additional sensors are needed in key monitoring areas. The vision sensors can be positioned according to the length and shape of the tunnel to ensure that surface defects inside the tunnel can be captured.

[0022] Based on the above analysis, the final sensor deployment scheme is as follows: Front of the vehicle: 2 vision sensors (forward and on both sides of the front). Rear of the vehicle: 2 vision sensors (rear-view and on both sides of the rear). Sides of the vehicle: 2 vision sensors (left and right, specifically for image acquisition on curved sections). Inside the vehicle: To prevent excessive sensors from occupying interface space or power, no additional equipment will be deployed inside the vehicle, maximizing the use of sensor interface space.

[0023] For example, the quantitative layout calculation and dynamic adjustment mechanism of the sensor array is as follows: Assuming the vehicle width is W and the vehicle height is H, the overlap rate of the target detection blind spots set at both ends and sides of the inspection vehicle is ƞ (usually ƞ is greater than or equal to 15%). Let the horizontal field of view of the vision sensor be α, the vertical field of view be β, and the pitch angle between the camera's principal optical axis and the horizontal plane be θ. The installation height and viewing angle are calculated, with the sensor installation height h as... c =H - Δh, where Δh is the safe clearance distance, taken as 0.2m~0.5m. Track surface coverage distance D cover The installation height meets the following requirements: .

[0024] Baseline distance and number of sensors are calculated, assuming the minimum curve radius of the inspection route is R. min When driving on a curve, the vehicle body shearing causes a lateral field of view shift of [amount missing]. Where L is the vehicle length.

[0025] To eliminate lateral blind spots, the lateral mounting offset distance of the lateral vision sensor is set. And the number of side cameras N side Determined by the following formula: .

[0026] The dynamic adaptive adjustment mechanism involves the vehicle continuously collecting data on the road curvature C and vehicle speed v in real time using an onboard gyroscope and line speed sensor. Triggering condition: When the real-time curvature C is greater than or equal to a preset curvature (e.g., a curve radius less than 300m), the sensor is triggered to adaptively adjust. The adjustment method involves dynamically adjusting the pitch angle of the cameras on both sides using a dual-axis gimbal. ,in, For vehicle speed, It is the acceleration due to gravity. The radius of the curve is given. The camera sampling frequency is increased simultaneously. Ensure that the image overlay rate is not less than 20% on highways or sharp curves.

[0027] Through this reasonable deployment, we ensured that the power supply of the vehicles was not overloaded. At the same time, by utilizing the limited sensor interface locations, we ensured that the railway inspection vehicles could obtain complete and accurate data collection throughout the inspection process, meeting the needs of different inspection areas (such as straight lines, curves, slopes, and special locations).

[0028] Taking into account limitations in power supply and the number of interfaces, as well as the basic parameters of the vehicles and the specific conditions of the inspection lines, this comprehensive deployment ensures that the sensors can operate efficiently and comprehensively cover key areas of the railway inspection lines. Through this integrated approach, railway inspection vehicles can provide high-quality infrastructure inspection data, guaranteeing the safety of railway operations.

[0029] Step S200: Use the vision sensor array to acquire surface images of the infrastructure of the target inspection route, and obtain a set of infrastructure surface images; In one possible embodiment, during railway inspection, a visual sensor array mounted on the inspection vehicle will capture images of the railway infrastructure from different angles, acquiring high-resolution image data in different inspection areas. The infrastructure surface image set includes surface information of the infrastructure, such as the contact surface of the track, the condition of the ballast bed, and the structural condition of tunnels. This image data will reflect the condition of the infrastructure, especially surface defects such as cracks, corrosion, and wear. During the inspection, the visual sensors continuously capture images based on the vehicle's travel path and speed, obtaining detailed image information from different locations. The acquired infrastructure surface image set will be used for defect identification in subsequent steps.

[0030] The purpose of this step is to acquire detailed surface data of the infrastructure along the target inspection route through efficient image acquisition, thereby providing data support for defect identification and structural health assessment.

[0031] Step S300: Obtain a set of images of surface defects of historical infrastructure and a set of identification results of surface defects of historical infrastructure to construct a surface defect identification prototype and obtain a set of surface defect identification prototypes; Furthermore, by acquiring a set of historical infrastructure surface defect images and a set of historical infrastructure surface defect identification results, a surface defect identification prototype is constructed to obtain a surface defect identification prototype set. In this embodiment, step S300 further includes: Obtain a set of images of surface defects on historical infrastructure and a set of identification results of surface defects on historical infrastructure, wherein each image of surface defects on historical infrastructure corresponds to one identification result of surface defects on historical infrastructure. Feature vectors are extracted from the set of images of surface defects of historical infrastructure to obtain a set of feature vectors for surface defects of historical infrastructure. The results of the identification of surface defects of historical infrastructure are aggregated into K clusters to obtain a set of K clustered results of surface defect identification of historical infrastructure. The feature vector sets of surface defect images of historical infrastructure are mapped and matched to obtain K clustered sets of feature vectors of surface defect images of historical infrastructure, where K is a positive integer. A prototype set for surface defect recognition is constructed based on the set of feature vectors of surface defect images of K clustered historical infrastructure and the set of surface defect recognition results of K clustered historical infrastructure.

[0032] In one embodiment, the historical infrastructure surface defect image set refers to infrastructure surface image data collected during past railway inspections. These images may contain different types of defects, such as cracks and corrosion, providing a historical dataset encompassing various situations. The historical infrastructure surface defect identification result set consists of the defect identification results corresponding to each historical infrastructure surface defect image, typically including the defect type (e.g., cracks, corrosion, damage) and the specific location of the defect in the image. These results are used to guide subsequent defect identification and classification work. Each historical infrastructure surface defect image corresponds to one historical infrastructure surface defect identification result.

[0033] This application overcomes the drawback of deep learning classifiers requiring retraining of the entire network when facing rare new defects in railway sites with small sample sizes by iteratively optimizing neighborhood density in a high-dimensional vector space, thus achieving rapid incremental online updates of new defect prototypes. Specifically, feature extractors are used to extract feature vectors from the set of historical infrastructure surface defect images, thereby obtaining the set of feature vectors for the historical infrastructure surface defect images. The K clustered historical infrastructure surface defect identification result sets are obtained by grouping the historical infrastructure surface defect identification results using a clustering algorithm. K is a positive integer representing the number of clusters. Each cluster set contains historical infrastructure surface defect identification results with similar defect features. Images and identification results with similar defect features are grouped using clustering algorithms (such as K-means, DBSCAN, etc.).

[0034] Preferably, the feature extractor in this application is constructed based on a feedforward neural network, specifically using a lightweight residual network (ResNet-18) as the backbone network architecture. The input receives a three-channel surface image tensor with a fixed size of 224×224×3; the backbone contains one 7×7 convolutional layer (stride 2), one 3×3 max pooling layer (stride 2), and four residual blocks with channel numbers of 64, 128, 256, and 512 respectively; the residual blocks are followed by a global mean pooling layer (GAP) to compress the two-dimensional feature map into a one-dimensional scalar; finally, a fully connected layer outputs a high-dimensional normalized semantic feature vector of D=128 dimensions.

[0035] The training data consists of 12,000 images of infrastructure surfaces covering various typical railway defects, including rail spalling, micro-cracks (0.1–0.5 mm wide), fastener anomalies, ballast deformation, and corrosion. These images are divided into a training set (9,600 images) and a validation set (2,400 images) in an 8:2 ratio. The optimizer used is Adam, with an initial learning rate of 10. -4 The batch size is set to 32, and the cross-entropy loss converges to the preset threshold of 0.01 to obtain the trained feature extractor.

[0036] For example, based on engineering experience or data distribution density, the number of clusters K is pre-specified by those skilled in the art, for instance, K=5, corresponding to track surface spalling, cracks, fastener anomalies, track bed deformation, and foreign object intrusion, respectively. K initial representative points are randomly selected from the multidimensional attribute feature space or selected based on density distribution as the initial centers of the K cluster sets. Each data point in the historical identification result set is traversed, and the geometric distance between that point and the K initial center points in the feature space is calculated. According to the principle of proximity allocation, the data point is assigned to the cluster set represented by the nearest center point. After all historical defect identification results have been assigned, for the obtained K preliminary cluster sets, the mean value of all member data points within each set is calculated across all attributes, i.e., the new geometric centroid of the set is located, and this mean point is set as the new center point of the cluster set. The two steps of sample assignment and cluster center update are repeated. As iteration progresses, the assignment status of each data point gradually stabilizes, and the positional variation of the cluster center points decreases. The iteration terminates when the positions of all cluster centers no longer move, or the moving distance is less than a preset minimum threshold.

[0037] The detection method based on clustering prototypes proposed in this application can achieve or even surpass the accuracy of traditional deep learning methods. Comparative experiments were conducted on a test set containing 3,000 real-world on-site inspection images. The evaluation metrics included accuracy, recall, F1-score, and mean precision. The comparison results are shown in Table 1.

[0038] Table 1 Comparison of Detection Accuracy

[0039] Each cluster set contains images of defects that are similar in the feature space and their corresponding recognition results. For example, all images that appear as cracks may be clustered into one category, while images that appear as corrosion may be clustered into another. This process helps to extract common features of different defect types. Based on the one-to-one mapping relationship between historical infrastructure surface defect images and their recognition results, the feature vector sets of these historical infrastructure surface defect images are mapped and matched to obtain the K clustered sets of feature vectors for historical infrastructure surface defect images. Using the clustering results and combining the features of various defect types, a prototype set for surface defect recognition is constructed. This prototype will serve as the basis for subsequent defect recognition tasks, helping automated systems quickly identify identical or similar defects in new images.

[0040] The algorithm in this application is deployed on an embedded edge computing terminal mounted on a railway inspection vehicle, with a power consumption limit of 50W, and processes 1920×1080 resolution images. A comparison of the computational efficiency metrics of each algorithm is shown in Table 2.

[0041] Table 2 Comparison of Calculation Efficiency Indicators

[0042] End-to-end deep learning networks require complex multi-scale region proposals or dense anchor box calculations across the entire image. This application simplifies the online matching process to lightweight feature extraction and cosine similarity calculation between a 128-dimensional vector and the prototype set. This application achieves a single-frame processing time of only 6.8ms, a processing frame rate of up to 147 FPS, and a GPU memory footprint of only 0.4GB, fully meeting the engineering requirements for real-time concurrent acquisition and detection of millimeter-level defects on the rail surface by railway inspection vehicles operating at high speeds of 120km / h (approximately 33.3m / s).

[0043] By analyzing historical data, a prototype set for surface defect recognition is constructed to identify defects in new images, thereby improving the accuracy and efficiency of defect detection.

[0044] Furthermore, based on the set of K clustered historical infrastructure surface defect image feature vectors and the set of K clustered historical infrastructure surface defect identification results, a surface defect identification prototype set is constructed. In this embodiment, step S300 further includes: Extract the first set of feature vectors for surface defects of historical infrastructure and the first set of identification results for surface defects of historical infrastructure from the set of feature vectors for surface defects of surface defects of surface infrastructure in the K clusters and the set of identification results for surface defects of surface infrastructure in the K clusters; Semantic integration is performed on the first clustered set of historical infrastructure surface defect identification results to obtain the first integrated surface defect identification result; Vector set analysis is performed on the feature vector set of surface defect images of the first clustered historical infrastructure to determine the feature vector of the first integrated surface defect image. The first integrated surface defect identification result and the first integrated surface defect image feature vector are associated to construct a first surface defect identification prototype; Prototype identification is performed on the K-1 clustered historical infrastructure surface defect image feature vector sets and the K-1 clustered historical infrastructure surface defect identification result sets to obtain K-1 surface defect identification prototypes. Combined with the first surface defect identification prototypes, a surface defect identification prototype set is obtained.

[0045] Furthermore, vector aggregation analysis is performed on the first clustered set of surface defect image feature vectors of historical infrastructure to determine the first integrated surface defect image feature vector. In this embodiment, step S300 further includes: Randomly extract one feature vector of surface defect images of historical infrastructure from the first cluster of feature vector sets of historical infrastructure surface defect images as the initial center vector; The similarity between the first clustered set of feature vectors of surface defects on historical infrastructure and the initial center vector is calculated, and the calculation results are filtered based on a preset similarity threshold to obtain the neighborhood of the initial center vector. A first cluster of historical infrastructure surface defect image feature vectors is randomly extracted from the edge of the initial center vector neighborhood as the iterative center vector, and the iterative center vector neighborhood is obtained. Determine whether the neighborhood density of the iterative center vector neighborhood is greater than or equal to the neighborhood density of the initial center vector neighborhood. If so, randomly extract a first clustered historical infrastructure surface defect image feature vector from the edge of the initial center vector neighborhood again, and iteratively update the iterative center vector until the number of iterations is satisfied to obtain the target center vector. Use the target center vector as the first integrated surface defect image feature vector.

[0046] Furthermore, by counting the total number of feature vectors of surface defect images of historical infrastructure in the first cluster of the initial center vector neighborhood, and comparing the statistical result with twice the preset similarity threshold mentioned above, the neighborhood density of the initial center vector is obtained.

[0047] In one possible embodiment, a first set of feature vectors for surface defects of historical infrastructure and a first set of identification results for surface defects of historical infrastructure are extracted from the set of feature vectors for surface defects of K clustered historical infrastructure and the set of identification results for surface defects of K clustered historical infrastructure. Furthermore, since the descriptions of different defect identification results may be different, but the defect types they express are similar, the first integrated surface defect identification results are obtained by integrating the first set of identification results for historical infrastructure by using a semantic integrator, so as to facilitate the standardization of defect detection results.

[0048] Preferably, a training set is obtained by using a set of multiple sample clustering results for identifying surface defects of historical infrastructure and a set of multiple sample integrated surface defect identification results obtained after integration and summarization of the multiple sample clustering results for identifying surface defects of historical infrastructure by those skilled in the art. The training set is then used to supervise the training of the framework built on the convolutional neural network until it converges, thus obtaining a trained semantic integrator.

[0049] In one possible embodiment, the feature vector that best represents the type of surface defect is determined by performing vector set analysis on the first clustered set of historical infrastructure surface defect image feature vectors, which is the first integrated surface defect image feature vector.

[0050] Preferably, a first clustered historical infrastructure surface defect image feature vector is randomly extracted from the first clustered historical infrastructure surface defect image feature vector set as the initial center vector. Then, using the cosine similarity calculation formula, the similarity between the first clustered historical infrastructure surface defect image feature vector set and the initial center vector is calculated iteratively. The calculation results are then filtered according to a preset similarity threshold set by those skilled in the art, and the first clustered historical infrastructure surface defect image feature vectors with similarity greater than or equal to the preset similarity threshold are added to the neighborhood of the initial center vector.

[0051] Then, from the edge of the initial center vector neighborhood, that is, from the two first clustered historical infrastructure surface defect image feature vectors corresponding to the maximum and minimum similarity values ​​of the initial center vector, a first clustered historical infrastructure surface defect image feature vector is randomly extracted as the iterative center vector, and the iterative center vector neighborhood is obtained in the same way as the initial center vector neighborhood.

[0052] The neighborhood density of the initial center vector is obtained by statistically analyzing the total number of feature vectors of surface defects in historical infrastructure images within the neighborhood of the initial center vector and comparing the statistical result with twice the preset similarity threshold. The neighborhood density of the initial center vector reflects the degree to which the initial center vector can represent the overall situation of the first cluster of feature vectors of surface defects in historical infrastructure images; a higher neighborhood density indicates that the initial center vector is more representative of the overall situation.

[0053] Furthermore, based on the same principle as obtaining the neighborhood density of the initial center vector, the neighborhood density of the iterative center vector neighborhood is obtained. Then, it is determined whether the neighborhood density of the iterative center vector neighborhood is greater than or equal to the neighborhood density of the initial center vector neighborhood. If so, a first clustered historical infrastructure surface defect image feature vector is randomly extracted from the edge of the initial center vector neighborhood again, and the iterative center vector is iteratively updated until the number of iterations preset by those skilled in the art is satisfied, and the target center vector is obtained. The target center vector is used as the first integrated surface defect image feature vector.

[0054] Step S400: Use the surface defect recognition prototype set to match and recognize the infrastructure surface image set to obtain the surface defect recognition result.

[0055] Furthermore, by using the surface defect identification prototype set to match and identify the infrastructure surface image set to obtain surface defect identification results, step S400 of this embodiment further includes: Feature vectors are extracted from the set of infrastructure surface images to obtain a set of infrastructure surface image feature vectors; The surface defect identification prototype set is used to identify the feature vector set of the infrastructure surface image to obtain the surface defect identification result.

[0056] Furthermore, step S400 in this embodiment of the application also includes: Multiple sample infrastructure surface images and feature vectors of multiple sample infrastructure surface images are obtained as training data; The training data is divided into a training set and a validation set. The training set is used to supervise the training of the framework built on the feedforward neural network, and the validation set is used to validate the trained framework. The network parameters of the framework are updated according to the validation results until the requirements are met, and the trained feature extractor is obtained. The feature extractor is used to extract feature vectors from the set of infrastructure surface images to obtain the set of infrastructure surface image feature vectors.

[0057] In one possible embodiment, the surface defect identification results are obtained after a model-based identification process based on the extracted feature vectors and identification prototypes. These results typically include the defect category (e.g., cracks, corrosion) and the specific location or severity of the defect. The surface defect identification prototype set is pre-built using algorithms based on historical data and known defect patterns for rapid identification of surface defects in images. The infrastructure surface image feature vector set is used to reflect important information in the image (e.g., edges, texture, color, etc.).

[0058] Preferably, a feature extractor is used to collect multiple sample infrastructure surface images containing various typical track, tunnel, and track bed surface conditions, such as smoothness, microcracks, severe spalling, and corrosion. These images are then combined with manually labeled or standard database images to generate corresponding sample infrastructure surface image feature vectors as training data. The training data is divided into a training set and a validation set according to a pre-set ratio, such as 8:2 or 7:3. The training set is used for forward computation and backward gradient update of model parameters, while the validation set is used to evaluate the model's generalization ability and convergence status on unseen samples.

[0059] A feature extractor framework based on a feedforward neural network is constructed. Its network structure, from input to output, includes: an input layer receiving a fixed-size three-channel surface image tensor; a multi-layer convolution and downsampling module consisting of multiple sequentially connected 2D convolutional layers, batch normalization layers, Rectified Linear Unit (ReLU) activation layers, and max-pooling layers, used to extract geometric edges, line directions, local textures, and semantic features of deep defects from the surface image layer by layer; a global feature aggregation layer using global average pooling (GAP) to compress the spatial 2D size of the deep feature map into a 1D scalar, eliminating the interference of spatial location on features; and a fully connected output layer that outputs a high-dimensional floating-point feature vector of a specified dimension through linear mapping and bias accumulation.

[0060] Before training, the weight matrices of each convolutional kernel and the weights of the fully connected layers in the network are randomly initialized using a normal distribution, and the bias terms are uniformly initialized to zero. In each training epoch, sample infrastructure surface images from the training set are batched into the feature extractor framework, sequentially passing through the feature transformations of each convolutional layer, activation layer, and pooling layer. Finally, the fully connected layer outputs the predicted infrastructure surface image feature vector. The predicted feature vector is compared with the corresponding sample target feature vector, and the difference loss value in the numerical dimension between the two is calculated, such as mean squared error or cross-entropy loss. The gradient of each network layer parameter is derived layer by layer from the output layer to the input layer using the backpropagation algorithm. An optimizer, such as the Adam optimizer or stochastic gradient descent optimizer, is used to iteratively update the weights of each convolutional kernel, batch normalized scaling and translation factor, and the weight matrix of the fully connected layers in the network based on the calculated gradient values, gradually reducing the difference between the predicted vector and the target vector.

[0061] After each training epoch, sample images from the validation set are input into the network in the current training state, and the average loss value and feature matching accuracy on the validation set are calculated. The training process is monitored based on the validation results: if the validation set loss value continues to decrease, training continues; if the validation set loss value no longer decreases or increases over several consecutive epochs, it indicates that the model is overfitting, and an early stopping mechanism is triggered, or the learning rate is adjusted by decay. When the model reaches the preset maximum number of training epochs, the loss value converges to the preset minimum threshold, or the validation set accuracy meets the expected requirements, training is terminated, the optimal weight parameters of the network at this point are saved, and the trained feature extractor is obtained.

[0062] Furthermore, the set of feature vectors for the infrastructure surface images is compared with the integrated surface defect image feature vectors contained in each surface defect identification prototype in the set of surface defect identification prototypes. Similarity is then calculated using the cosine similarity formula, and the surface defect identification prototype corresponding to the maximum similarity in the similarity identification results is taken as the matching surface defect identification prototype. Finally, the integrated surface defect identification results from the matching surface defect identification prototypes are taken as the surface defect identification results. This enables the rapid and accurate identification of potential defects on the surface of railway infrastructure through efficient image processing and machine learning methods. By using a pre-constructed prototype set of surface defect identification and feature vector extraction, the accuracy and speed of identification can be improved, ensuring the timely detection of potential safety hazards.

[0063] In summary, the multi-view vision-based railway infrastructure defect detection method provided in this application has the following beneficial effects: This application acquires basic information about railway inspection vehicles, deploys sensors along the target inspection line to obtain a completed visual sensor array, and then uses this array to collect surface images of the infrastructure along the target inspection line, resulting in a set of infrastructure surface images. Furthermore, it acquires a set of historical infrastructure surface defect images and a set of historical infrastructure surface defect identification results to construct a surface defect identification prototype set. Finally, it uses this prototype set to match and identify the infrastructure surface images, obtaining the surface defect identification results. This achieves the technical effect of improving the reliability and accuracy of defect detection.

[0064] Example 2, as Figure 2 As shown, based on the same inventive concept as Embodiment 1 above, this application provides a railway infrastructure defect detection system based on multi-view vision, the system comprising: The visual sensor array acquisition module 11 is used to acquire basic information of railway inspection vehicles, combine the target inspection line to deploy sensors, and obtain the deployed visual sensor array. Infrastructure surface image set acquisition module 12 is used to acquire surface images of the infrastructure of the target inspection route using the vision sensor array, and obtain infrastructure surface image set; The prototype construction module 13 is used to acquire a set of historical infrastructure surface defect images and a set of historical infrastructure surface defect identification results to construct a surface defect identification prototype and obtain a surface defect identification prototype set. The surface defect identification result acquisition module 14 is used to match and identify the infrastructure surface image set using the surface defect identification prototype set to obtain the surface defect identification result.

[0065] Furthermore, the prototype identification module 13 is used to perform the following steps: Feature vectors are extracted from the set of images of surface defects of historical infrastructure to obtain a set of feature vectors for surface defects of historical infrastructure. The results of the identification of surface defects of historical infrastructure are aggregated into K clusters to obtain a set of K clustered results of surface defect identification of historical infrastructure. The feature vector sets of surface defect images of historical infrastructure are mapped and matched to obtain K clustered sets of feature vectors of surface defect images of historical infrastructure, where K is a positive integer. A prototype set for surface defect recognition is constructed based on the set of feature vectors of surface defect images of K clustered historical infrastructure and the set of surface defect recognition results of K clustered historical infrastructure.

[0066] Furthermore, the prototype identification module 13 is used to perform the following steps: Extract the first set of feature vectors for surface defects of historical infrastructure and the first set of identification results for surface defects of historical infrastructure from the set of feature vectors for surface defects of surface defects of surface infrastructure in the K clusters and the set of identification results for surface defects of surface infrastructure in the K clusters; Semantic integration is performed on the first clustered set of historical infrastructure surface defect identification results to obtain the first integrated surface defect identification result; Vector set analysis is performed on the feature vector set of surface defect images of the first clustered historical infrastructure to determine the feature vector of the first integrated surface defect image. The first integrated surface defect identification result and the first integrated surface defect image feature vector are associated to construct a first surface defect identification prototype; Prototype identification is performed on the K-1 clustered historical infrastructure surface defect image feature vector sets and the K-1 clustered historical infrastructure surface defect identification result sets to obtain K-1 surface defect identification prototypes. Combined with the first surface defect identification prototypes, a surface defect identification prototype set is obtained.

[0067] Furthermore, the prototype identification module 13 is used to perform the following steps: Randomly extract one feature vector of surface defect images of historical infrastructure from the first cluster of feature vector sets of historical infrastructure surface defect images as the initial center vector; The similarity between the first clustered set of feature vectors of surface defects on historical infrastructure and the initial center vector is calculated, and the calculation results are filtered based on a preset similarity threshold to obtain the neighborhood of the initial center vector. A first cluster of historical infrastructure surface defect image feature vectors is randomly extracted from the edge of the initial center vector neighborhood as the iterative center vector, and the iterative center vector neighborhood is obtained. Determine whether the neighborhood density of the iterative center vector neighborhood is greater than or equal to the neighborhood density of the initial center vector neighborhood. If so, randomly extract a first clustered historical infrastructure surface defect image feature vector from the edge of the initial center vector neighborhood again, and iteratively update the iterative center vector until the number of iterations is satisfied to obtain the target center vector. Use the target center vector as the first integrated surface defect image feature vector.

[0068] Furthermore, by counting the total number of feature vectors of surface defect images of historical infrastructure in the first cluster of the initial center vector neighborhood, and comparing the statistical result with twice the preset similarity threshold mentioned above, the neighborhood density of the initial center vector is obtained.

[0069] Furthermore, the surface defect identification result acquisition module 14 is used to perform the following steps: Feature vectors are extracted from the set of infrastructure surface images to obtain a set of infrastructure surface image feature vectors; The surface defect identification prototype set is used to identify the feature vector set of the infrastructure surface image to obtain the surface defect identification result.

[0070] Furthermore, the surface defect identification result acquisition module 14 is used to perform the following steps: Multiple sample infrastructure surface images and feature vectors of multiple sample infrastructure surface images are obtained as training data; The training data is divided into a training set and a validation set. The training set is used to supervise the training of the framework built on the feedforward neural network, and the validation set is used to validate the trained framework. The network parameters of the framework are updated according to the validation results until the requirements are met, and the trained feature extractor is obtained. The feature extractor is used to extract feature vectors from the set of infrastructure surface images to obtain the set of infrastructure surface image feature vectors.

[0071] Furthermore, the vision sensor array acquisition module 11 is used to perform the following steps: The basic information is extracted according to a preset set of basic parameters to obtain the set of basic parameters; The basic information is extracted according to the preset deployment configuration indicators to obtain the number of sensor interface locations and power supply capacity; The target inspection route is analyzed according to the preset route indicators to obtain the inspection route indicators. The visual sensor array is obtained by combining the number of sensor interface locations and power supply capacity with the basic parameter set and the inspection line indicators.

[0072] Furthermore, the preset set of basic parameters includes vehicle type, number of carriages, vehicle length, vehicle height, and vehicle width.

[0073] Through the foregoing detailed description of the railway infrastructure defect detection method based on multi-view vision, those skilled in the art can clearly understand that the railway infrastructure defect detection system based on multi-view vision in this embodiment, as corresponding to the system disclosed in Embodiment 2, has corresponding functional modules and beneficial effects as it corresponds to the method disclosed in Embodiment 1. For relevant details, please refer to the description in the method section.

[0074] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for detecting defects in railway infrastructure based on multi-view vision, characterized in that, The method includes: Obtain basic information about railway inspection vehicles, deploy sensors in conjunction with the target inspection line, and obtain the completed visual sensor array. The visual sensor array is used to acquire surface images of the infrastructure of the target inspection route, thereby obtaining a set of infrastructure surface images. A set of images and identification results of surface defects on historical infrastructure are obtained to construct a surface defect identification prototype set. The surface defect identification prototype set is used to match and identify the infrastructure surface image set to obtain the surface defect identification result; This involves acquiring a set of historical infrastructure surface defect images and a set of historical infrastructure surface defect identification results to construct a surface defect identification prototype set, which includes: Feature vectors are extracted from the set of images of surface defects of historical infrastructure to obtain a set of feature vectors for surface defects of historical infrastructure. The results of the identification of surface defects of historical infrastructure are aggregated into K clusters to obtain a set of K clustered results of surface defect identification of historical infrastructure. The feature vector sets of surface defect images of historical infrastructure are mapped and matched to obtain K clustered sets of feature vectors of surface defect images of historical infrastructure, where K is a positive integer. A prototype set for surface defect identification is constructed based on the set of feature vectors of surface defect images of K clustered historical infrastructure and the set of surface defect identification results of K clustered historical infrastructure. This involves acquiring basic information about railway inspection vehicles, deploying sensors along the target inspection line, and obtaining a completed visual sensor array, including: The basic information is extracted according to a preset set of basic parameters to obtain the set of basic parameters; The basic information is extracted according to the preset deployment configuration indicators to obtain the number of sensor interface locations and power supply capacity; The target inspection route is analyzed according to the preset route indicators to obtain the inspection route indicators. The visual sensor array is obtained by combining the sensor interface location and number and power supply capacity with the basic parameter set and the inspection line indicators; The preset set of basic parameters includes vehicle type, number of carriages, vehicle length, vehicle height, and vehicle width.

2. The railway infrastructure defect detection method based on multi-view vision as described in claim 1, characterized in that, Based on the K clustered sets of historical infrastructure surface defect image feature vectors and the K clustered sets of historical infrastructure surface defect identification results, a surface defect identification prototype set is constructed, including: Extract the first set of feature vectors for surface defects of historical infrastructure and the first set of identification results for surface defects of historical infrastructure from the set of feature vectors for surface defects of surface defects of surface infrastructure in the K clusters and the set of identification results for surface defects of surface infrastructure in the K clusters; Semantic integration is performed on the first clustered set of historical infrastructure surface defect identification results to obtain the first integrated surface defect identification result; Vector set analysis is performed on the feature vector set of surface defect images of the first clustered historical infrastructure to determine the feature vector of the first integrated surface defect image. The first integrated surface defect identification result and the first integrated surface defect image feature vector are associated to construct a first surface defect identification prototype; Prototype identification is performed on the K-1 clustered historical infrastructure surface defect image feature vector sets and the K-1 clustered historical infrastructure surface defect identification result sets to obtain K-1 surface defect identification prototypes. Combined with the first surface defect identification prototypes, a surface defect identification prototype set is obtained.

3. The railway infrastructure defect detection method based on multi-view vision as described in claim 2, characterized in that, Vector set analysis is performed on the feature vector set of surface defect images of the first clustered historical infrastructure to determine the feature vector of the first integrated surface defect image, including: Randomly extract one feature vector of surface defect images of historical infrastructure from the first cluster of feature vector sets of historical infrastructure surface defect images as the initial center vector; The similarity between the first clustered set of feature vectors of surface defects on historical infrastructure and the initial center vector is calculated, and the calculation results are filtered based on a preset similarity threshold to obtain the neighborhood of the initial center vector. A first cluster of historical infrastructure surface defect image feature vectors is randomly extracted from the edge of the initial center vector neighborhood as the iterative center vector, and the iterative center vector neighborhood is obtained. Determine whether the neighborhood density of the iterative center vector neighborhood is greater than or equal to the neighborhood density of the initial center vector neighborhood. If so, randomly extract a first clustered historical infrastructure surface defect image feature vector from the edge of the initial center vector neighborhood again, and iteratively update the iterative center vector until the number of iterations is satisfied to obtain the target center vector. Use the target center vector as the first integrated surface defect image feature vector.

4. The railway infrastructure defect detection method based on multi-view vision as described in claim 3, characterized in that, The neighborhood density of the initial center vector is obtained by counting the total number of feature vectors of surface defects of historical infrastructure in the first cluster of the neighborhood of the initial center vector and comparing the statistical result with twice the preset similarity threshold.

5. The railway infrastructure defect detection method based on multi-view vision as described in claim 1, characterized in that, The surface defect identification prototype set is used to match and identify the infrastructure surface image set to obtain surface defect identification results, including: Feature vectors are extracted from the set of infrastructure surface images to obtain a set of infrastructure surface image feature vectors; The surface defect identification prototype set is used to identify the feature vector set of the infrastructure surface image to obtain the surface defect identification result.

6. The railway infrastructure defect detection method based on multi-view vision as described in claim 5, characterized in that, include: Multiple sample infrastructure surface images and feature vectors of multiple sample infrastructure surface images are obtained as training data; The training data is divided into a training set and a validation set. The training set is used to supervise the training of the framework built on the feedforward neural network, and the validation set is used to validate the trained framework. The network parameters of the framework are updated according to the validation results until the requirements are met, and the trained feature extractor is obtained. The feature extractor is used to extract feature vectors from the set of infrastructure surface images to obtain the set of infrastructure surface image feature vectors.

7. A railway infrastructure defect detection system based on multi-view vision, characterized in that, The system is used to execute the railway infrastructure defect detection method based on multi-view vision as described in any one of claims 1-6, including: The visual sensor array acquisition module is used to acquire basic information of railway inspection vehicles, combine it with the target inspection line to deploy sensors, and obtain the completed visual sensor array. The infrastructure surface image set acquisition module is used to acquire surface images of the infrastructure of the target inspection route using the vision sensor array, and obtain an infrastructure surface image set. The prototype construction module is used to acquire a set of images of surface defects on historical infrastructure and a set of identification results of surface defects on historical infrastructure to construct a surface defect identification prototype and obtain a set of surface defect identification prototypes. The surface defect identification result acquisition module is used to match and identify the infrastructure surface image set using the surface defect identification prototype set to obtain the surface defect identification result.

Images