Rail surface defect detection method, system, electronic device and readable storage medium

Abstract

The application relates to a rail surface defect detection method and system, an electronic device and a readable storage medium, and belongs to the technical field of rail detection. The method comprises the following steps: acquiring a plurality of rail surface images and point cloud data; splicing the plurality of rail surface images to obtain an analysis image; generating a three-dimensional image of the rail according to the point cloud data; inputting the analysis image into a deep learning convolutional neural network for identification to determine defect information in the analysis image; if there is a defect region whose defect category is not determined, the defect region whose defect category is not determined is determined as a region to be classified, a speculation strategy is determined according to the positional relationship between the region to be classified and other defect regions, and the defect information of the region to be classified is determined according to the speculation strategy; and the identified analysis image and the three-dimensional image are matched according to corresponding positions, and the defect information is marked in the three-dimensional image. The application improves the work efficiency of the polishing process after the defects are identified.

Description

Technical Field

[0001] This application relates to the technical field of rail inspection, and in particular to a method, system, electronic device and readable storage medium for detecting surface defects in rails. Background Technology

[0002] With the rapid development of railway transportation, train speeds and load-bearing capacity are constantly increasing, placing higher demands on the quality and safety of rails. Rails are a critical infrastructure component of railway transportation. When abnormalities occur in rails, such as surface cracks (whether present or absent), fish-scale patterns, spalling, scratches on the top surface, thickened edges, corrosion, or corrugation, they can pose significant safety hazards to railway operations. Therefore, timely safety inspections of rail facilities are essential.

[0003] Visual inspection of rail surface defects can be achieved by capturing images using a linear array camera mounted on the underside of a train, followed by image processing methods to detect and identify the defects. Therefore, designing and implementing an automatic rail surface defect detection method is of great significance, and achieving precise defect localization and accurate shape segmentation is crucial for the success of such an automatic method.

[0004] However, the actual situation of rail surface defects is more complex, and the current image processing-based rail defect detection methods are not very resistant to interference. This may result in some defects not being accurately identified as their corresponding types, requiring further judgment by staff, which affects the efficiency of the subsequent grinding process. Summary of the Invention

[0005] To improve the efficiency of the grinding process after defect identification, this application provides a method, system, electronic device, and readable storage medium for detecting defects on the surface of rails.

[0006] Firstly, this application provides a method for detecting defects on the surface of rails, employing the following technical solution:

[0007] Acquire multiple rail surface images and point cloud data;

[0008] Multiple images of the rail surface are stitched together to obtain an analysis image;

[0009] A three-dimensional image of the rail is generated based on the point cloud data;

[0010] The analyzed image is input into a deep learning convolutional neural network for recognition to determine the defect information in the analyzed image, wherein the defect information includes the defect region and the defect category to which the defect region belongs;

[0011] If there are defect areas whose defect category is not determined, then the defect areas whose defect category is not determined are identified as areas to be classified. An inference strategy is determined based on the positional relationship between the areas to be classified and other defect areas. The defect information of the areas to be classified is determined based on the inference strategy.

[0012] The identified analysis image is matched with the three-dimensional image according to the corresponding positions, and the defect information is marked in the three-dimensional image.

[0013] By employing the above technical solution, multiple rail surface images are stitched together to obtain an analysis image with a wider field of view. This facilitates a comprehensive observation of the rail surface condition, reducing omissions caused by image field-of-view limitations. A 3D image of the rail is generated based on point cloud data, providing 3D morphological information of the rail surface. This helps to more intuitively understand the geometric features and surface condition of the rail. Deep learning convolutional neural networks can automatically extract features from the analysis image and identify defect areas and their respective defect categories, greatly improving the efficiency and accuracy of defect identification. For defect areas whose defect categories are not determined, a positional relationship inference strategy is used to determine their defect information, which helps reduce omissions and misjudgments, improving the comprehensiveness of identification. The defect information in the identified analysis image is marked in the 3D image, making the defect location more intuitive and clear in 3D space. This allows staff to initially determine the specific defects in each defect area and carry out targeted processing, improving defect processing efficiency.

[0014] Furthermore, the step of determining the inference strategy based on the positional relationship between the region to be classified and other defect regions includes:

[0015] Determine a preset number of neighboring regions that are closest to the region to be classified;

[0016] Determine the positional relationship between the region to be classified and each of the adjacent regions;

[0017] If the positional relationship is such that the region to be classified is not connected to any of the adjacent regions, then the first inference strategy is adopted.

[0018] If the positional relationship is such that the region to be classified is adjacent to any of the adjacent regions, then the second inference strategy is adopted.

[0019] Furthermore, the first inference strategy includes:

[0020] Each of the adjacent regions is assigned a first category.

[0021] If there is only one type of the first category, then the defect category located in the first category will be determined as the defect category of the area to be classified.

[0022] If the first category has more than one type, then it includes:

[0023] Determine the probability level of the existence of consecutive defects in each of the first categories;

[0024] The adjacent regions are sorted in descending order of the probability levels of each of the first categories to obtain a first sequence;

[0025] Obtain the distance between the region to be classified and each of the adjacent regions;

[0026] The adjacent regions are sorted in ascending order of the distances to obtain a second sequence;

[0027] Determine the first order of the adjacent regions in the first sequence and the second order in the second sequence, and calculate the average of the first order and the second order according to a preset ratio;

[0028] The adjacent regions are sorted in ascending order of the mean to obtain a third sequence;

[0029] The category of the adjacent region located at the head of the third sequence is determined as the defect category of the region to be classified.

[0030] Furthermore, the second inference strategy includes:

[0031] Each of the adjacent regions is identified as a first category, and the probability level of continuous defects in each of the first categories is determined.

[0032] The adjacent regions are sorted in descending order of the probability levels of each of the first categories to obtain a first sequence;

[0033] The first category located in the adjacent region at the head of the first sequence is determined as the first candidate category;

[0034] The region to be classified is divided into multiple sub-regions;

[0035] Each of the sub-regions is input into a deep learning convolutional neural network for recognition, and the second category to which each of the sub-regions belongs is determined.

[0036] Determine whether there is more than one second category corresponding to all the sub-regions; if not, determine that the second category is the defect category of the region to be classified.

[0037] If so, then it includes:

[0038] Determine whether the probability level of consecutive defects in adjacent areas is greater than or equal to the preset level;

[0039] If it is greater than, then the defect category of the area to be compared is determined to be the combination of the defect category of the adjacent areas and all the second categories;

[0040] If it is not greater than, then determine the proportion of each second category to the total number of sub-regions;

[0041] The defect category of the region to be compared is determined to be a combination of the second categories, and the region to be compared is marked according to the proportion corresponding to each of the second categories.

[0042] Furthermore, the step of stitching together multiple rail surface images to obtain an analysis image includes:

[0043] The rail surface image is filtered, enhanced, and noise-reduced to obtain a preprocessed rail surface image.

[0044] Arrange the preprocessed rail surface images according to the shooting order;

[0045] Each of the preprocessed rail surface images is processed to obtain a grayscale image;

[0046] Identify the overlapping regions in two adjacent grayscale images, overlap the regions in two adjacent preprocessed rail surface images, delete the overlapping region in one of the rail surface images and stitch them together to obtain the analysis image.

[0047] Further, determining the overlapping region in two adjacent grayscale images includes:

[0048] The contrast regions in two adjacent grayscale images are determined respectively, and the contrast regions are located close to each other in the two adjacent grayscale images;

[0049] First contrast data of a predetermined number of columns is determined from the side of the contrast region near the edge of one of the grayscale images;

[0050] In the contrast region of another grayscale image, along a preset direction, find second contrast data that matches the first contrast data;

[0051] The first comparison data and the second comparison data are identified as overlapping areas. The first comparison data is moved away from the edge, and the process of searching for second comparison data that matches the first comparison data in the comparison area of ​​another grayscale image along a preset direction is repeated until no second comparison data that matches the first data is found.

[0052] Furthermore, before inputting the analyzed image into a deep learning convolutional neural network for recognition, the method further includes:

[0053] The analysis image is identified by applying a rail recognition neural network model to determine the rail portion and background portion of the analysis image;

[0054] The rail portion image is binarized to obtain the pixels of the rail surface portion. Based on the pixel coordinates of the rail surface portion, the rail portion image is divided into a rail surface image and a rail body image.

[0055] The track surface image and the track body image are labeled in the analysis image so that the deep learning convolutional neural network can identify defects on the track surface and track body, respectively.

[0056] Secondly, this application provides a rail surface defect detection device, comprising:

[0057] The data acquisition module is used to acquire multiple rail surface images and point cloud data;

[0058] An image stitching module is used to stitch together multiple images of the rail surface to obtain an analytical image;

[0059] A 3D image generation module is used to generate a 3D image of the rail based on the point cloud data;

[0060] The defect analysis module is used to input the analysis image into a deep learning convolutional neural network for recognition and to determine the defect information in the analysis image, wherein the defect information includes the defect region and the defect category to which the defect region belongs;

[0061] The defect inference module is used to determine the defect area whose defect category is not determined as a region to be classified if there is a defect area whose defect category is not determined. The module determines an inference strategy based on the positional relationship between the region to be classified and other defect areas, and determines the defect information of the region to be classified based on the inference strategy.

[0062] The labeling module is used to match the identified analysis image with the three-dimensional image according to the corresponding positions, and to label the defect information in the three-dimensional image.

[0063] Thirdly, this application provides an electronic device that adopts the following technical solution:

[0064] An electronic device, comprising:

[0065] At least one processor;

[0066] Memory;

[0067] At least one computer program, wherein the at least one computer program is stored in the memory and configured to be executed by the at least one processor, the at least one computer program being configured to: perform the method as described in any one of the first aspects.

[0068] Fourthly, this application provides a computer-readable storage medium, which adopts the following technical solution:

[0069] A computer-readable storage medium storing a computer program that can be loaded by a processor and execute the method as described in any one of the first aspects.

[0070] In summary, this application includes at least one of the following beneficial technical effects:

[0071] 1. By stitching together multiple images of the rail surface, an analytical image with a wider field of view can be obtained, which helps to comprehensively observe the condition of the rail surface and reduce omissions caused by the limitation of the image field of view;

[0072] 2. Generate a 3D image of the rail based on point cloud data to provide 3D morphological information of the rail surface, which helps to understand the geometric features and surface condition of the rail more intuitively;

[0073] 3. Deep learning convolutional neural networks can automatically extract features from analyzed images and identify defective regions and their respective defect categories, greatly improving the efficiency and accuracy of defect identification;

[0074] 4. For defect areas whose defect category is not determined, the defect information can be determined by inferring their location relationship. This helps to reduce omissions and misjudgments, improve the comprehensiveness of identification, and provide a reference for staff to make judgments.

[0075] 5. Mark the defect information in the identified analysis image in the 3D image, making the defect location more intuitive and clear in 3D space. This allows staff to initially determine the specific defects in each defect area and carry out targeted treatment, thereby improving the efficiency of defect handling. Attached Figure Description

[0076] Figure 1 This is a flowchart illustrating the rail surface defect detection method in the embodiments of this application.

[0077] Figure 2 This is a structural block diagram of the rail surface defect detection device in the embodiments of this application.

[0078] Figure 3 This is a structural block diagram of the electronic device in the embodiments of this application. Detailed Implementation

[0079] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0080] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article, unless otherwise specified, generally indicates that the preceding and following related objects have an "or" relationship.

[0081] This application discloses a method for detecting surface defects in steel rails. (Refer to...) Figure 1 This is performed by an electronic device, which can be a server or a terminal device. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The terminal device can be a smartphone, tablet, desktop computer, etc., but is not limited to these. (Steps S101 to S106)

[0082] Step S101: Acquire multiple rail surface images and point cloud data.

[0083] Specifically, the rail surface defect detection equipment includes a line array camera mounted on an inspection vehicle. At least one line array camera is installed at the bottom of the inspection vehicle, and the arrangement of the line array cameras is perpendicular to the length extension direction of the rail. A lidar system is also installed on the inspection vehicle, which emits a laser beam while the vehicle is moving. The laser beam scans the scene and measures the area, and the reflected laser beam is received by a receiver to calculate the three-dimensional coordinate information of the object's surface.

[0084] Furthermore, both the line scan camera and the lidar are connected to electronic equipment, which then acquires multiple images and point cloud data of the rail surface. The rail surface images are obtained by continuous shooting from the line scan camera, while the point cloud data is obtained by the lidar.

[0085] Step S102: Stitch together multiple rail surface images to obtain an analysis image. This includes (steps S1021 to S1024):

[0086] Step S1021: Preprocess the rail surface image to obtain the preprocessed rail surface image.

[0087] Specifically, due to the influence of the on-site environment of the rails, the captured images of the rail surface may be unclear due to environmental interference. Therefore, it is necessary to process the rail surface images. The processing includes noise reduction, contrast enhancement, and brightness adjustment to improve image quality. Invalid areas in the rail surface images, such as shadows and reflections, are identified and removed.

[0088] Step S1022: Arrange the pre-processed rail surface images according to the shooting order.

[0089] Specifically, the electronic device arranges the pre-processed rail surface images in sequence.

[0090] Step S1023: Perform grayscale value processing on each preprocessed rail surface image to obtain a grayscale image.

[0091] Specifically, the electronic equipment performs grayscale processing on the pre-processed rail surface image, which can reduce the impact of factors such as color on the defect identification process.

[0092] Step S1024: Determine the overlapping areas in two adjacent grayscale images, overlap the overlapping areas of two adjacent preprocessed rail surface images, delete the overlapping area in one of the rail surface images and stitch them together to obtain the analysis image.

[0093] Specifically, to ensure the integrity of the image data, adjacent rail surface images partially overlap. The electronic equipment can be configured with a line scan camera to take pictures at preset time intervals to obtain a rail surface image. Therefore, when the inspection vehicle travels at a preset constant speed, the overlapping area in each pair of adjacent rail surface images is the same size. The electronic equipment can then directly delete one of the overlapping areas and stitch the other rail surface image onto it, and so on, to obtain the analysis image.

[0094] However, if the inspection vehicle is starting or stopping, and its speed is less than the preset speed, the overlapping areas in adjacent rail surface images captured during this period will be inconsistent in size. Therefore, it is necessary to compare and determine the overlapping areas, including (steps S11 to S15):

[0095] Step S11: Determine the contrast regions in two adjacent grayscale images respectively. The contrast regions are located close to each other in the two adjacent grayscale images.

[0096] Specifically, the electronic device pre-defines a contrast region at each end of each grayscale image, so the contrast regions of two adjacent grayscale images are located close to each other. For example, the contrast region of the current grayscale image that is close to the contrast region of the next grayscale image, and the contrast region of the next grayscale image that is close to the contrast region of the current grayscale image, are considered a close set of contrast regions.

[0097] Step S13: Determine the first contrast data of a preset number of columns from the side of the contrast region near the edge of one of the grayscale images.

[0098] Specifically, in order to accurately compare consistent data in two adjacent grayscale images, the electronic device needs to search step by step. Therefore, it selects the first comparison data from the comparison area for comparison. Since a grayscale image is composed of pixels arranged in rows and columns, the electronic device can select the first comparison data of a preset number of columns.

[0099] Step S14: In the contrast area of ​​another grayscale image, along a preset direction, find the second contrast data that matches the first contrast data.

[0100] Specifically, the preset direction can be either away from the edge of the grayscale image or close to the edge of the grayscale image. The electronic device searches for data that matches the first comparison data along the preset direction and determines it as the second comparison data.

[0101] Step S15: Determine the first comparison data and the second comparison data as the overlapping area, move the first comparison data away from the edge, and repeat the step of searching for the second comparison data that matches the first comparison data in the comparison area of ​​another grayscale image along a preset direction until no second comparison data that matches the first data is found.

[0102] Specifically, after the electronic device finds the overlapping area with the first comparison data, it moves the first comparison data so that it is adjacent to the previous comparison data, and then repeats step S14 to further expand the overlapping area. When no second comparison data matching the first comparison data is found, step S15 is stopped, and the search for the overlapping area is completed.

[0103] Step S103: Generate a 3D image of the rail based on the point cloud data.

[0104] Specifically, due to measurement errors or environmental interference, point cloud data contains outliers and anomalies. Electronic devices remove these outliers, and filtering algorithms are used to reduce the impact of noise on subsequent processing, thereby converting the point cloud data into a format suitable for further processing. A mesh generation method is then applied to transform the point cloud data into a 3D mesh model, i.e., a 3D image.

[0105] Step S104: Input the analysis image into a deep learning convolutional neural network for recognition and determine the defect information in the analysis image, wherein the defect information includes the defect region and the defect category of the defect region.

[0106] Specifically, the electronic device pre-trains a deep learning convolutional neural network model. The electronic device sets up training data, which includes a large number of input sample images with surface defects on the rails, and output sample images labeled with the defects and their respective categories. The electronic device divides the training data into a training set, a validation set, and a test set, with corresponding proportions of 70%, 20%, and 10%.

[0107] Forward propagation: The training data is passed through the network to the output layer to calculate the predicted values; Loss calculation: The loss function is used to calculate the difference between the predicted and true values; Backpropagation: Based on the gradient of the loss function, the network propagates backward layer by layer to the input layer to calculate the gradient of each parameter; Parameter update: The network parameters are updated using optimization algorithms, such as stochastic gradient descent (SGD) and Adam, to reduce the loss; Iterative update: The process of forward propagation, loss calculation, backpropagation, and parameter update is repeated until the specified number of iterations or error threshold is reached.

[0108] The model's performance is evaluated using a validation set, and the model structure and parameters are adjusted based on the evaluation results. The final performance of the model is evaluated using a test set to ensure good generalization ability. Finally, the electronic device trains the deep learning convolutional neural network model.

[0109] The electronic device inputs the image to a deep learning convolutional neural network to determine the defect information in the image, including the defect region and the corresponding defect type.

[0110] Furthermore, after the electronic device inputs the analyzed image into a deep learning convolutional neural network for recognition, it can determine the defect region and the corresponding defect category. However, since the actual defects on the rail surface may be more complex—for example, multiple defects may be superimposed in the same location, the defect manifestation may fall between two categories, or the degree of defect on the rail surface may be minor, resulting in indistinct defect features in the image—it may be difficult for the deep learning convolutional neural network model to identify the specific defect type. Therefore, if there are defect regions whose defect category has not been determined, steps S105 and S106 are executed; otherwise, step S106 is executed directly.

[0111] Step S105: Determine the defect area whose defect category has not been determined as the area to be classified, determine the inference strategy based on the positional relationship between the area to be classified and other defect areas, and determine the defect information of the area to be classified based on the inference strategy.

[0112] The process of determining the inference strategy based on the positional relationship between the area to be classified and other defect areas includes (steps S21 to S24):

[0113] Step S21: Determine the preset number of neighboring regions that are closest to the region to be classified.

[0114] Specifically, the electronic device takes the area to be classified as the center, determines the distance between each surrounding defect area and the center, arranges them in order of increasing distance, and selects a preset number of defect areas as adjacent areas.

[0115] Step S22: Determine the positional relationship between the area to be classified and each adjacent area.

[0116] Specifically, positional relationships include both adjacent and non-adjacent relationships.

[0117] Step S23: If the positional relationship is that the region to be classified is not connected to any of its adjacent regions, then the first inference strategy is adopted.

[0118] Step S24: If the positional relationship is that the region to be classified is adjacent to any neighboring region, then the second inference strategy is adopted.

[0119] The first prediction strategy includes (steps S231 to S239):

[0120] Step S231: Determine the category corresponding to each adjacent area as the first category.

[0121] If there is only one type in the first category, then proceed to step S432: determine the defect category located in the first category as the area to be classified.

[0122] Specifically, if there is only one type of first category in the areas adjacent to the area to be classified, it can be inferred that the defect category of the area to be classified is the same as that of the surrounding areas.

[0123] If the first category has more than one type, then it includes:

[0124] Step S233: Determine the probability level of continuous defects in each first category.

[0125] Specifically, the electronic device identifies the category corresponding to each adjacent area as the first category, and further determines the probability level of continuous defects in the first category. The probability level is roughly divided into three levels: low, medium, and high.

[0126] Different types of defects have different levels of probability of consecutive defects occurring.

[0127] For example, fish-scale patterns on rails are typically caused by fine cracks and spalling resulting from fatigue wear on the wheel-rail contact surface, and their shape resembles fish scales. This defect usually manifests as a localized phenomenon and is unlikely to appear continuously across the entire rail surface. Therefore, the probability of fish-scale patterns on rails being a continuous defect is low.

[0128] Rail surface abrasion is caused by sliding friction between the wheel and rail during locomotive starting or braking. This type of abrasion is often localized and shallow, making it unlikely to form a continuous defect. Therefore, the possibility of continuous defects arising from rail surface abrasion is low.

[0129] Rail surface cracks can be caused by a variety of factors, such as material defects, welding defects, wheel-rail contact stress concentration, and fatigue accumulation. Under specific conditions, such as on curved sections, heavy-load lines, or sections with frequent braking, cracks may develop continuously along the longitudinal or transverse direction of the rail. The probability of continuous rail surface cracks is moderate.

[0130] Rail spalling typically develops from cracks or fatigue damage on the rail surface. When the cracks extend to a certain extent, the material between the cracks detaches under the viscous force of the train wheels, forming spalling. In severely worn or fatigued sections, spalling may occur continuously. The likelihood of continuous rail spalling is moderate.

[0131] Rail edge thickening is caused by excessive friction between the wheel and the rail, resulting in wear, deformation, or bulging at the rail edge. On curved sections or heavy-load lines, the lateral pressure from the wheel on the rail is enhanced, making edge thickening more severe and potentially causing it to occur continuously along the longitudinal direction of the rail. The likelihood of continuous rail edge thickening is moderate.

[0132] Rail corrosion occurs due to a chemical reaction between the rail surface and moisture and oxygen in the air. Corrosion is particularly severe in humid or rainy areas. The corrosion spreads continuously along the rail surface, forming large areas of rust. Therefore, continuous corrosion of rails is highly likely.

[0133] Rail corrugation is a wavy wear pattern on the rail surface caused by the combined effects of various factors, including coupled vibrations of the vehicle-track system and wheel-rail feedback vibrations. This wear occurs continuously along the longitudinal direction of the rail and may cover the entire rail surface. Therefore, rail corrugation is highly likely to occur continuously.

[0134] Step S234: Sort the corresponding adjacent regions in descending order of the probability of each first category to obtain the first sequence.

[0135] Therefore, if there are 10 adjacent regions, and the first category of the 10 adjacent regions is rail top surface scratch, rail edge thickening, and rail corrosion, the electronic device will sort the adjacent regions in the order of rail corrosion, rail edge thickening, and top surface scratch to obtain the first sequence.

[0136] Step S235: Obtain the distance between the region to be classified and each of its neighboring regions.

[0137] Specifically, the electronic device establishes a coordinate system in the analyzed image, thereby obtaining the coordinates of the center of the region to be classified and the coordinates of the center points of each adjacent region, and calculating the Euclidean distance between the coordinates of each center point.

[0138] Step S236: Sort the corresponding adjacent regions in ascending order of each distance to obtain the second sequence.

[0139] Specifically, the defects in the region to be classified may be clustered, and therefore may be closer to the first category of the neighboring regions. To facilitate inference, a second sequence is obtained by sorting.

[0140] Step S237: Determine the first order of adjacent regions in the first sequence and the second order in the second sequence, and calculate the average of the first order and the second order according to a preset ratio.

[0141] Specifically, the sum of the preset ratios corresponding to the first and second sequences is 1, and the preset ratios are set by the electronic device according to actual needs. If the successive probability has a greater impact on the inferred defect category, the preset ratio corresponding to the first sequence is greater than the preset ratio corresponding to the second sequence; conversely, if the distance to adjacent areas has a greater impact on the inferred defect category, the preset ratio corresponding to the second sequence is greater than the preset ratio corresponding to the first sequence.

[0142] Therefore, the electronic device calculates the average of the first and second orders of adjacent regions at a preset ratio.

[0143] Step S238: Sort the corresponding adjacent regions according to the ascending order of the mean to obtain the third sequence.

[0144] Step S239: Determine the category of the adjacent region located at the head of the third sequence as the defect category of the region to be classified.

[0145] Specifically, the smaller the mean of adjacent regions in the third sequence, the higher the similarity between the category of the adjacent region and the defect category of the region to be classified, considering both the probability of defect continuity and distance. Therefore, the adjacent region at the head of the third sequence is more similar to the defect category of the region to be classified, and the electronic device can infer that it is the defect category of the region to be classified.

[0146] Furthermore, the second inference strategy includes (steps S241 to S240):

[0147] Step S241: Determine the category corresponding to each adjacent area as the first category, and determine the probability level of continuous defects in each first category.

[0148] Step S242: Sort the corresponding adjacent regions in descending order of the probability levels of each first category to obtain the first sequence.

[0149] Specifically, the electronic device performs steps S241 and S242 in the same way as steps S231, S233 and S234.

[0150] Step S243: Determine the first category of the adjacent region located at the head of the first sequence as the first candidate category.

[0151] Specifically, the adjacent area at the head of the first sequence has the highest probability of having consecutive defects, so its first category is determined as the first candidate category.

[0152] Step S244: Divide the region to be classified into multiple sub-regions.

[0153] Specifically, electronic devices can use a grid segmentation method to divide the area to be classified into multiple sub-regions.

[0154] Step S445: Input each sub-region into the deep learning convolutional neural network for recognition and determine the second category to which each sub-region belongs.

[0155] Specifically, if the region to be classified consists of multiple types of defects superimposed, then after segmentation, the deep learning convolutional neural network may be able to identify the defect category corresponding to the sub-region and determine it as the second category.

[0156] Step S246: Determine whether there is more than one second category corresponding to all sub-regions; if not, proceed to step S247: Determine the second category as the defect category of the region to be classified.

[0157] Specifically, if there is only one second category for all sub-regions, it may be because the defect features of the region to be classified are not obvious, thus preventing its identification. Therefore, the defect category of the region to be classified in the second category is inferred.

[0158] If so, then it includes:

[0159] Step S248: Determine whether the probability level of consecutive defects in adjacent areas is greater than or equal to the preset level.

[0160] If it is greater than, then proceed to step S249: determine the defect category of the area to be compared as a combination of the defect category of the adjacent areas and all second categories.

[0161] Specifically, if the preset level can be medium, then when the probability level of consecutive defects in adjacent areas is medium or high, the area to be compared has a high probability of consecutive defects. Therefore, it is inferred that the defect category of the adjacent areas is a combination of the second category.

[0162] If it is not greater than, then proceed to step S240: determine the proportion of each second category to the total number of sub-regions, determine the defect category of the region to be compared as a combination of the second categories, and mark the regions to be compared according to the proportion corresponding to each second category.

[0163] Specifically, when the predicted comparison area is a combination of the second category, the proportion of each category is marked for easy viewing by staff.

[0164] Step S106: Match the identified analysis image with the three-dimensional image according to the corresponding positions, and mark the defect information in the three-dimensional image.

[0165] Specifically, the electronic device presets the relative positions of the analysis image and the 3D image, then matches the identified analysis image with the 3D image, marking the defect information in the 3D image. The electronic device's interface design allows for adjustments to the color scheme and control shapes, and it can display defective tracks within the system. Staff can view the defect location and type in the 3D image.

[0166] Electronic devices can separately label defect regions identified by deep convolutional neural networks and defect regions whose categories are inferred.

[0167] Once defects are detected in the track, staff can re-verify the defects based on the data displayed in the 3D images, paying particular attention to areas where electronic equipment has inferred defects. After verification, the rails are ground using grinding equipment to meet railway regulations. This process involves repeatedly acquiring track surface images and point cloud data, comparing the ground track with relevant data from standard tracks to determine if the grinding has met standards.

[0168] In another possible implementation, before inputting the analyzed image into a deep learning convolutional neural network for recognition, the analyzed image can be further segmented to improve the recognition speed. The above method also includes (steps S31 to S33):

[0169] Step S31: Apply the rail recognition neural network model to identify and analyze the image, and determine the rail portion and background portion of the image.

[0170] Specifically, the electronic device pre-trains a rail recognition neural network model. It takes an input image as input and outputs a result that can identify the rails in the image. Then, the electronic device inputs the analyzed image into the rail recognition neural network model. The output analyzed image has the rail portion marked. Therefore, the electronic device can segment the analyzed image based on the markings to obtain the rail portion image, while the remaining portion is the background image.

[0171] Step S32: Binarize the rail section image to obtain the pixels of the rail surface. Based on the pixel coordinates of the rail surface, divide the rail section image into a rail surface image and a rail body image.

[0172] Specifically, because the track surface is closer to the line scan camera, it is brighter under illumination. The electronic equipment first determines a threshold. If a pixel's grayscale value is greater than the threshold, its grayscale value is set to 255 (white), indicating that the pixel is part of the track surface. If a pixel's grayscale value is less than or equal to the threshold, its grayscale value is set to 0 (black), indicating that the pixel does not belong to the track surface. After binarization, the pixels with a grayscale value of 255 are identified as the track surface pixels, and the image corresponding to each pixel is thus identified as the track surface image, while the remaining portion is the track body image.

[0173] Step S33: Divide the analysis image into track surface image and track body image so that the deep learning convolutional neural network can identify defects on the track surface and track body respectively.

[0174] Specifically, the electronic device inputs the track surface image and the track body image into the deep learning convolutional neural network model, and the model can then identify defects on the track surface and the track body respectively.

[0175] To better implement the above method, this application also provides a rail surface defect detection device, referring to... Figure 2 The rail surface defect detection device 200 includes:

[0176] Data acquisition module 201 is used to acquire multiple rail surface images and point cloud data;

[0177] The image stitching module 202 is used to stitch together multiple rail surface images to obtain an analysis image;

[0178] The 3D image generation module 203 is used to generate a 3D image of the rail based on point cloud data;

[0179] The defect analysis module 204 is used to input the analysis image into a deep learning convolutional neural network for recognition and to determine the defect information in the analysis image. The defect information includes the defect region and the defect category to which the defect region belongs.

[0180] The defect inference module 205 is used to determine the defect area whose defect category is not determined as the area to be classified if there is a defect area whose defect category is not determined. The module determines the inference strategy based on the positional relationship between the area to be classified and other defect areas, and determines the defect information of the area to be classified based on the inference strategy.

[0181] The labeling module 206 is used to match the identified analysis image with the three-dimensional image according to the corresponding positions, and to label the defect information in the three-dimensional image.

[0182] Furthermore, the defect estimation module 205 determines the estimation strategy based on the positional relationship between the area to be classified and other defect areas, specifically for:

[0183] Determine the preset number of neighboring regions closest to the area to be classified;

[0184] Determine the positional relationship between the area to be classified and each adjacent area;

[0185] If the location relationship is such that the region to be classified is not adjacent to any of its neighboring regions, then the first inference strategy is adopted.

[0186] If the location relationship is such that the region to be classified is adjacent to any neighboring region, then the second inference strategy is adopted.

[0187] Furthermore, the first prediction strategy of the defect prediction module 205 is specifically used for:

[0188] The category corresponding to each adjacent region is determined as the first category;

[0189] If there is only one type in the first category, then the defect category located in the first category will be determined as the defect category of the area to be classified.

[0190] If the first category has more than one type, then it includes:

[0191] Determine the probability level of consecutive defects in each of the first categories;

[0192] The adjacent regions are sorted in descending order of the probability levels of each first category to obtain the first sequence;

[0193] Obtain the distances between the region to be classified and each of its adjacent regions;

[0194] Sort the corresponding adjacent regions in ascending order of their distances to obtain the second sequence;

[0195] Determine the first order of adjacent regions in the first sequence and the second order in the second sequence, and calculate the average of the first order and the second order according to a preset ratio;

[0196] Sort the corresponding adjacent regions in ascending order of their mean to obtain the third sequence;

[0197] The category of the adjacent region at the head of the third sequence is determined as the defect category of the region to be classified.

[0198] Furthermore, the second inference strategy of the defect inference module 205 is specifically used for:

[0199] Each adjacent region is designated as a first category, and the probability level of continuous defects in each first category is determined.

[0200] The adjacent regions are sorted in descending order of the probability levels of each first category to obtain the first sequence;

[0201] The first category in the adjacent region at the head of the first sequence is determined as the first candidate category;

[0202] Divide the area to be classified into multiple sub-areas;

[0203] Each sub-region is input into a deep learning convolutional neural network for recognition, and the second category to which each sub-region belongs is determined.

[0204] Determine whether there is more than one second category for all sub-regions; if not, determine the second category as the defect category of the region to be classified.

[0205] If so, then it includes:

[0206] Determine whether the probability level of consecutive defects in adjacent areas is greater than or equal to the preset level;

[0207] If it is greater than, then the defect category of the area to be compared is determined to be the combination of the defect category of the adjacent areas and all the second categories;

[0208] If it is not greater than, then determine the proportion of each second category to the total number of sub-regions;

[0209] The defect categories of the areas to be compared are determined to be combinations of the second categories, and the areas to be compared are marked according to the proportion corresponding to each second category.

[0210] Furthermore, the image stitching module 202 is specifically used for:

[0211] The rail surface image is filtered, enhanced, and noise is eliminated to obtain a preprocessed rail surface image;

[0212] Arrange the pre-processed rail surface images in the order they were taken;

[0213] Each preprocessed rail surface image is processed to obtain a grayscale image;

[0214] Identify the overlapping regions in two adjacent grayscale images, overlap the regions of two adjacent preprocessed rail surface images, delete the overlapping region in one of the rail surface images, and then stitch them together to obtain the analysis image.

[0215] Furthermore, the image stitching module 202 analyzes and determines the overlapping regions in two adjacent grayscale images, specifically for:

[0216] Identify the contrast regions in two adjacent grayscale images, where the contrast regions are located close to each other in the two adjacent grayscale images;

[0217] First contrast data of a predetermined number of columns is determined from the side of the contrast region near the edge of one of the grayscale images;

[0218] In the contrast area of ​​another grayscale image, along a preset direction, find second contrast data that matches the first contrast data;

[0219] The first comparison data and the second comparison data are identified as overlapping areas. The first comparison data is moved away from the edge, and the process of searching for the second comparison data that matches the first comparison data in the comparison area of ​​another grayscale image along a preset direction is repeated until no second comparison data that matches the first data is found.

[0220] Furthermore, it also includes:

[0221] The segmentation module is used to apply a rail recognition neural network model to identify and analyze images, and to determine the rail portion and the background portion of the analyzed image.

[0222] The binarization module is used to binarize the rail section image to obtain the pixels of the rail surface. Based on the coordinates of the pixels of the rail surface, the rail section image is divided into the rail surface image and the rail body image.

[0223] The labeling module is used to label the track surface image and track body image in the analysis image so that the deep learning convolutional neural network can identify defects on the track surface and track body respectively.

[0224] The various variations and specific examples of the methods in the foregoing embodiments are also applicable to the rail surface defect detection device of this embodiment. Through the foregoing detailed description of the rail surface defect detection method, those skilled in the art can clearly understand the implementation method of the rail surface defect detection device of this embodiment. Therefore, for the sake of brevity, it will not be described in detail here.

[0225] To better implement the above methods, embodiments of this application provide an electronic device, referring to... Figure 3The electronic device 300 includes a processor 301, a memory 303, and a display screen 305. The memory 303 and the display screen 305 are both connected to the processor 301, such as via a bus 302. Optionally, the electronic device 300 may also include a transceiver 304. It should be noted that in practical applications, the transceiver 304 is not limited to one type, and the structure of this electronic device 300 does not constitute a limitation on the embodiments of this application.

[0226] Processor 301 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 301 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0227] Bus 302 may include a pathway for transmitting information between the aforementioned components. Bus 302 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 302 may be divided into address bus, data bus, control bus, etc.

[0228] The memory 303 may be a ROM (Read Only Memory) or other type of static storage device capable of storing static information and instructions, RAM (Random Access Memory) or other type of dynamic storage device capable of storing information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0229] The memory 303 is used to store application code that executes the solution of this application, and its execution is controlled by the processor 301. The processor 301 is used to execute the application code stored in the memory 303 to implement the content shown in the foregoing method embodiments.

[0230] Figure 3 The electronic device 300 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0231] This application also provides a computer-readable storage medium storing a computer program. When executed by a processor, the program implements the rail surface defect detection method provided in the above embodiments, stitching together multiple rail surface images to obtain an analysis image with a wider field of view. This helps to comprehensively observe the condition of the rail surface, reducing omissions caused by image field of view limitations. A three-dimensional image of the rail is generated based on point cloud data, providing three-dimensional morphological information of the rail surface. This helps to more intuitively understand the geometric features and surface condition of the rail. Deep learning convolutional neural networks can automatically extract features from the analysis image and identify defect areas and their respective defect categories, greatly improving the efficiency and accuracy of defect identification. For defect areas whose defect categories are not determined, their defect information is determined through a positional relationship inference strategy, which helps to reduce omissions and misjudgments, improve the comprehensiveness of identification, and mark the defect information in the identified analysis image in the three-dimensional image, making the defect location more intuitive and clear in three-dimensional space. This allows staff to initially determine the specific defects in each defect area and carry out targeted processing, improving defect processing efficiency.

[0232] In this embodiment, the computer-readable storage medium can be a tangible device that holds and stores instructions used by an instruction execution device. The computer-readable storage medium can be, but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any combination thereof. Specifically, the computer-readable storage medium can be a portable computer disk, a hard disk, a USB flash drive, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), staging random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory stick, floppy disk, optical disk, magnetic disk, mechanical encoding device, or any combination thereof.

[0233] The computer program in this embodiment includes program code for performing all the aforementioned methods. The program code may include instructions corresponding to the method steps provided in the above embodiments. The computer program can be downloaded from a computer-readable storage medium to various computing / processing devices, or downloaded to an external computer or external storage device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network). The computer program can be executed entirely on the user's computer as a standalone software package.

[0234] The above are all preferred embodiments of this application, and are not intended to limit the scope of protection of this application. Therefore, all equivalent changes made in accordance with the structure, shape and principle of this application should be covered within the scope of protection of this application.

[0235] Additionally, it should be understood that relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

Claims

1. A method for detecting surface defects in steel rails, characterized in that, include: Acquire multiple rail surface images and point cloud data; Multiple images of the rail surface are stitched together to obtain an analysis image; A three-dimensional image of the rail is generated based on the point cloud data; The analyzed image is input into a deep learning convolutional neural network for recognition to determine the defect information in the analyzed image, wherein the defect information includes the defect region and the defect category to which the defect region belongs; If there are defect areas whose defect category is not determined, then the defect areas whose defect category is not determined are identified as areas to be classified. An inference strategy is determined based on the positional relationship between the areas to be classified and other defect areas. The defect information of the areas to be classified is determined based on the inference strategy. The identified analysis image is matched with the three-dimensional image according to the corresponding positions, and the defect information is marked in the three-dimensional image; The step of determining the inference strategy based on the positional relationship between the region to be classified and other defect regions includes: Determine a preset number of neighboring regions that are closest to the region to be classified; Determine the positional relationship between the region to be classified and each of the adjacent regions; If the positional relationship is such that the region to be classified is not connected to any of the adjacent regions, then the first inference strategy is adopted. If the positional relationship is such that the region to be classified is adjacent to any of the adjacent regions, then the second inference strategy is adopted. The second inference strategy includes: Each of the adjacent regions is identified as a first category, and the probability level of continuous defects in each of the first categories is determined. The adjacent regions are sorted in descending order of the probability levels of each of the first categories to obtain a first sequence; The first category located in the adjacent region at the head of the first sequence is determined as the first candidate category; The region to be classified is divided into multiple sub-regions; Each of the sub-regions is input into a deep learning convolutional neural network for recognition, and the second category to which each of the sub-regions belongs is determined. Determine whether there is more than one second category corresponding to all the sub-regions; if not, determine that the second category is the defect category of the region to be classified. If so, then it includes: Determine whether the probability level of consecutive defects in adjacent areas is greater than or equal to the preset level; If it is greater than, then the defect category of the area to be compared is determined to be the combination of the defect category of the adjacent areas and all the second categories; If it is not greater than, then determine the proportion of each second category to the total number of sub-regions; The defect category of the region to be compared is determined to be a combination of the second categories, and the region to be compared is marked according to the proportion corresponding to each of the second categories.

2. The method according to claim 1, characterized in that, The first inference strategy includes: Each of the adjacent regions is assigned a first category. If there is only one type of the first category, then the defect category located in the first category will be determined as the defect category of the area to be classified. If the first category has more than one type, then it includes: Determine the probability level of the existence of consecutive defects in each of the first categories; The adjacent regions are sorted in descending order of the probability levels of each of the first categories to obtain a first sequence; Obtain the distance between the region to be classified and each of the adjacent regions; The adjacent regions are sorted in ascending order of the distances to obtain a second sequence; Determine the first order of the adjacent regions in the first sequence and the second order in the second sequence, and calculate the average of the first order and the second order according to a preset ratio; The adjacent regions are sorted in ascending order of the mean to obtain a third sequence; The category of the adjacent region located at the head of the third sequence is determined as the defect category of the region to be classified.

3. The method according to claim 1, characterized in that, The step of stitching together multiple rail surface images to obtain an analysis image includes: The rail surface image is filtered, enhanced, and noise-reduced to obtain a preprocessed rail surface image. Arrange the preprocessed rail surface images according to the shooting order; Each of the preprocessed rail surface images is processed to obtain a grayscale image; Identify the overlapping regions in two adjacent grayscale images, overlap the overlapping regions in two adjacent preprocessed rail surface images, delete the overlapping region in one of the rail surface images and stitch them together to obtain the analysis image.

4. The method according to claim 3, characterized in that, Determining the overlapping region in two adjacent grayscale images includes: The contrast regions in two adjacent grayscale images are determined respectively, and the contrast regions are located close to each other in the two adjacent grayscale images; First contrast data of a predetermined number of columns is determined from the side of the contrast region near the edge of one of the grayscale images; In the contrast region of another grayscale image, along a preset direction, find second contrast data that matches the first contrast data; The first comparison data and the second comparison data are identified as overlapping areas. The first comparison data is moved away from the edge, and the process of searching for second comparison data that matches the first comparison data in the comparison area of ​​another grayscale image along a preset direction is repeated until no second comparison data that matches the first comparison data is found.

5. The method according to claim 1, characterized in that, Before inputting the analyzed image into a deep learning convolutional neural network for recognition, the method further includes: The analysis image is identified by applying a rail recognition neural network model to determine the rail portion and background portion of the analysis image; The rail portion image is binarized to obtain the pixels of the rail surface portion. Based on the pixel coordinates of the rail surface portion, the rail portion image is divided into a rail surface image and a rail body image. The track surface image and the track body image are labeled in the analysis image so that the deep learning convolutional neural network can identify defects on the track surface and track body, respectively.

6. A rail surface defect detection device, characterized in that, include: The data acquisition module is used to acquire multiple rail surface images and point cloud data; An image stitching module is used to stitch together multiple images of the rail surface to obtain an analytical image; A 3D image generation module is used to generate a 3D image of the rail based on the point cloud data; The defect analysis module is used to input the analysis image into a deep learning convolutional neural network for recognition and to determine the defect information in the analysis image, wherein the defect information includes the defect region and the defect category to which the defect region belongs; The defect inference module is used to determine the defect area whose defect category is not determined as a region to be classified if there is a defect area whose defect category is not determined. The module determines an inference strategy based on the positional relationship between the region to be classified and other defect areas, and determines the defect information of the region to be classified based on the inference strategy. The labeling module is used to match the identified analysis image with the three-dimensional image according to the corresponding positions, and to label the defect information in the three-dimensional image; The defect inference module determines an inference strategy based on the positional relationship between the region to be classified and other defect regions, specifically for: Determine the preset number of neighboring regions closest to the area to be classified; Determine the positional relationship between the area to be classified and each adjacent area; If the location relationship is such that the region to be classified is not adjacent to any of its neighboring regions, then the first inference strategy is adopted. If the location relationship is such that the region to be classified is adjacent to any neighboring region, then the second inference strategy is adopted. The second inference strategy of the defect inference module is specifically used for: Each adjacent region is designated as a first category, and the probability level of continuous defects in each first category is determined. The adjacent regions are sorted in descending order of the probability levels of each first category to obtain the first sequence; The first category in the adjacent region at the head of the first sequence is determined as the first candidate category; Divide the area to be classified into multiple sub-areas; Each sub-region is input into a deep learning convolutional neural network for recognition, and the second category to which each sub-region belongs is determined. Determine whether there is more than one second category for all sub-regions; if not, determine the second category as the defect category of the region to be classified. If so, then it includes: Determine whether the probability level of consecutive defects in adjacent areas is greater than or equal to the preset level; If it is greater than, then the defect category of the area to be compared is determined to be the combination of the defect category of the adjacent areas and all the second categories; If it is not greater than, then determine the proportion of each second category to the total number of sub-regions; The defect categories of the areas to be compared are determined to be combinations of the second categories, and the areas to be compared are marked according to the proportion corresponding to each second category.

7. An electronic device, characterized in that, include: At least one processor; At least one memory; At least one computer program, wherein the at least one computer program is stored in the memory and configured to be executed by the at least one processor, the at least one computer program being configured to perform the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer program is stored that can be loaded by a processor and executed as described in any one of claims 1 to 5.

Images