Rail image fastener recognition method and device based on structural prior

Abstract

The present specification relates to the technical field of rail fastener detection, and specifically discloses a rail image fastener identification method and device based on structural prior, wherein the method comprises: acquiring a target rail image and rail structure parameters; determining the position coordinates of the rail in the horizontal direction in the target rail image; selecting a detection region from the target rail image based on the position coordinates of the rail in the horizontal direction in the target rail image and the rail structure parameters, wherein the detection region comprises a fastener region; extracting a candidate region from the detection region, wherein the candidate region comprises a fastener candidate rectangular frame; and classifying the candidate region based on a neural network to identify the fastener from the candidate region. The above scheme can accurately realize rail fastener positioning and ensure the real-time requirement of high-speed railway fastener positioning identification.

Description

TECHNICAL FIELD

[0001] The present specification relates to the technical field of rail fastener detection, and in particular to a rail image fastener identification method and device based on structural priori. BACKGROUND

[0002] Steel rail fasteners are important parts that connect steel rails and sleepers. Once the fasteners are missing, broken, or incomplete, etc., the abnormal state of the fasteners will bring major hidden dangers to rail traffic safety, and even major safety accidents. In recent years, with the development of machine vision technology, rail inspection vehicles are used to take photos of the track, and image processing technology is used to automatically judge the state of the fasteners, which gradually replaces manual inspection and becomes a new trend.

[0003] The cross positioning method is the most widely used and mainstream method in fastener positioning methods. The fasteners have a distinct position feature on the ballast track, i.e., the fasteners are installed on the sleepers and are located at the intersection of the sleepers and the steel rails, and the two parts naturally form a cross structure on the track bed, which can be used as a reference to determine the relative position of the fasteners. In recent years, a large number of monolithic track beds have been used on high-speed rail and subway lines to replace traditional ballast tracks, which has brought about a huge change in structure. When the monolithic track bed is used for fastener positioning, only the relative coordinates of the steel rail in the fixed vertical direction are available, and the traditional fastener positioning method cannot be used for positioning.

[0004] Currently, many methods have achieved good detection and positioning effects for rail fastener detection technology: 1) HOG histogram is used to describe the features of the fasteners, and then the Hamming distance nearest neighbor algorithm is used to realize fastener positioning; 2) a method combining gray level and gradient is used to coarsely position the position of the sleeper, and then a template matching method is used to accurately position the fastener. However, the above algorithms use many feature calculations and template matching schemes, which result in an algorithm speed that cannot meet the real-time requirements of high-speed rail.

[0005] At present, there is no effective solution to the above problems. SUMMARY

[0006] The embodiments of the present specification provide a rail image fastener identification method and device based on structural priori to solve the problem that the fastener detection method in the prior art cannot meet the real-time requirements.

[0007] The embodiment of the specification provides a track image fastener recognition method, including: acquiring a target track image and a track structure parameter; determining a position coordinate of a track in the target track image in a horizontal direction, wherein the horizontal direction is a direction perpendicular to an extension direction of the track in the target track image; selecting a detection region from the target track image based on the position coordinate of the track in the target track image in the horizontal direction and the track structure parameter, wherein the detection region includes a fastener region; extracting a candidate region from the detection region, wherein the candidate region includes a fastener candidate rectangular frame; and classifying the candidate region based on a neural network to identify a fastener from the candidate region.

[0008] In one embodiment, determining the position coordinate of the track in the target track image in the horizontal direction includes: performing gray projection on the target track image along a vertical direction to determine the position coordinate of the track in the target track image in the horizontal direction, wherein the vertical direction is the extension direction of the track in the target track image.

[0009] In one embodiment, extracting the candidate region from the detection region includes: performing region binarization processing on the detection region to obtain a binarization-processed detection region; and performing gray projection on the binarization-processed detection region along the horizontal direction to extract the candidate region from the detection region.

[0010] In one embodiment, classifying the candidate region based on the neural network to identify the fastener from the candidate region includes: classifying the candidate region using a ResNet18 neural network to determine whether the fastener exists in the candidate region.

[0011] In one embodiment, the activation function in the ResNet18 neural network is a Mish activation function.

[0012] In one embodiment, a parallel structure of two CBM modules and an identity mapping channel is adopted in a residual module of the ResNet18 neural network.

[0013] In one embodiment, after the candidate region is extracted from the detection region, the method further includes: in a case where the number of fastener candidate rectangular frames in the candidate region does not satisfy a preset condition, acquiring a second track image adjacent to the target track image; splicing a preset part of the target track image and the second track image to obtain a spliced target image, and extracting a candidate region from the spliced target image.

[0014] The embodiment of the present specification further provides a track image fastener recognition device based on structure prior, comprising: an acquisition module configured to acquire a target track image and track structure parameters; a determination module configured to determine a horizontal position coordinate of a track in the target track image, wherein the horizontal direction is a direction perpendicular to an extension direction of the track in the target track image; a selection module configured to select a detection region from the target track image based on the horizontal position coordinate of the track in the target track image and the track structure parameters, wherein the detection region comprises a fastener region; an extraction module configured to extract a candidate region from the detection region, wherein the candidate region comprises a fastener candidate rectangular frame; and an identification module configured to classify the candidate region based on a neural network to identify a fastener from the candidate region.

[0015] The embodiment of the present specification further provides a track image fastener recognition device based on structure prior, which is an embedded device, comprising a CPU end and a GPU end; the CPU end is configured to: acquire a target track image and track structure parameters; determine a horizontal position coordinate of a track in the target track image; select a detection region from the target track image based on the horizontal position coordinate of the track in the target track image and the track structure parameters; extract a candidate region from the detection region; wherein the detection region comprises a fastener region, and the candidate region comprises a fastener candidate rectangular frame; and the GPU end is configured to: classify the candidate region based on a neural network to identify a fastener from the candidate region.

[0016] The embodiment of the present specification further provides a computer device, comprising a processor and a memory for storing processor-executable instructions, wherein the processor executes the instructions to implement the steps of the track image fastener recognition method based on structure prior described in any of the above embodiments.

[0017] The embodiment of the present specification further provides a computer-readable storage medium having computer instructions stored thereon, wherein the instructions are executed to implement the steps of the track image fastener recognition method based on structure prior described in any of the above embodiments.

[0018] In the embodiment of the present application, a track image fastener recognition method is provided, which can obtain a target track image and track structure parameters, determine the position coordinates of a track in the target track image in a horizontal direction, wherein the horizontal direction is a direction perpendicular to the extension direction of the track in the target track image, select a detection area from the target track image based on the position coordinates of the track in the target track image in the horizontal direction and the track structure parameters, wherein the detection area includes a fastener area; extract a candidate area from the detection area, wherein the candidate area includes a fastener candidate rectangular frame; and classify the candidate area based on a neural network to recognize the fastener from the candidate area. In the above scheme, after obtaining the target track image, the position of the track in the target track image can be determined, and based on the position of the track and the track structure parameters, a detection area including a fastener area can be selected, and a candidate area including a fastener candidate rectangular frame can be extracted from the detection area. By combining the track structure parameters and other prior knowledge to select the detection area, the efficiency and accuracy of the detection area picking can be improved, and the computational complexity can be reduced. Then, based on the neural network, the fastener can be recognized from the candidate area, which can further improve the recognition accuracy of the fastener, and determine whether the fastener exists in the candidate area. This not only accurately realizes the positioning of the track fastener, but also guarantees the real-time requirement of the high-speed railway for fastener positioning and recognition. The above scheme solves the problem that the fastener detection method in the prior art cannot meet the real-time requirement, and achieves the technical effect of effectively improving the fastener recognition efficiency while ensuring the recognition accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0019] The accompanying drawings, which are incorporated herein and form a part of the specification, illustrate the present application and, together with the description, further serve to explain the principles of the present application and to enable a person skilled in the art to make and use the present application. In the drawings:

[0020] Figure 1 A flowchart of a track image fastener recognition method based on structural prior in an embodiment of the present application is shown;

[0021] Figure 2 A raw image of a track image in an embodiment of the present application is shown;

[0022] Figure 3 A flowchart of a track image fastener recognition method based on structural prior in an embodiment of the present application is shown;

[0023] Figure 4 A detection area identification map of a track image in an embodiment of the present application is shown;

[0024] Figure 5 A region binary algorithm result of a detection area of a track image in an embodiment of the present application is shown;

[0025] Figure 6 An algorithm flow chart of region binarization of a detection region of a track image in an embodiment of the present specification is shown;

[0026] Figure 7 A curve graph of a Mish activation function in an embodiment of the present specification is shown;

[0027] Figure 8 An improved residual structure schematic diagram in an embodiment of the present specification is shown;

[0028] Figure 9 A schematic diagram of a fastener detection classification result in an embodiment of the present specification is shown;

[0029] Figure 10 A schematic diagram of a track image fastener recognition device based on structural prior in an embodiment of the present specification is shown;

[0030] Figure 11 A schematic diagram of a computer device in an embodiment of the present specification is shown. DETAILED DESCRIPTION

[0031] The principles and spirits of the present specification will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are given only to enable those skilled in the art to better understand and implement the present specification, and do not limit the scope of the present specification in any way. On the contrary, these embodiments are provided to make the present disclosure more transparent and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0032] Those skilled in the art know that the embodiments of the present specification can be implemented as a system, device, method or computer program product. Therefore, the present disclosure can be embodied in the form of a complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0033] The embodiment of the present specification provides a track image fastener recognition method based on structural prior. Figure 1A flow chart of a method for rail image fastener recognition based on structural prior in an embodiment of the present specification is shown. Although the present specification provides method operation steps or device structures as shown in the following embodiments or drawings, more or fewer operation steps or module units can be included in the method or device based on routine or non-creative labor. In steps or structures that do not have necessary causality in logic, the execution order of the steps or the module structure of the device is not limited to the execution order or module structure shown in the embodiment description and drawings of the present specification. When the method or module structure is applied to the actual device or terminal product, it can be sequentially executed or executed in parallel (for example, parallel processor or multi-thread processing environment, even distributed processing environment) according to the method or module structure shown in the embodiment or drawing.

[0034] Specifically, as shown in the present specification, an embodiment provides a method for rail image fastener recognition based on structural prior, which can include the following steps: Figure 1

[0035] Step S101, obtaining a target rail image and a rail structure parameter.

[0036] Specifically, the method in the embodiment of the present specification can be applied to a computer device. A target rail image can be obtained. The target rail image can be a rail image captured by a camera. The target rail image can include a rail, a tie and a fastener. Please refer to Figure 2 , which shows the original image of the rail image in an embodiment of the present specification. As shown in Figure 2 , the target rail image can include a rail 21, a tie 22 and a fastener 23. The extension direction of the rail 21 is the vertical direction.

[0037] In one embodiment, the rail structure parameter can include at least one of the following: length and width of the rail, size parameter of the fastener, size parameter of the tie, position of the tie relative to the rail, position of the fastener relative to the tie, position of the fastener relative to the rail, etc. The pixel number corresponding to the fastener and / or the tie in the rail image and the pixel number corresponding to the distance of the fastener and / or the tie relative to the rail in the rail image can be determined according to the rail structure parameters and the camera shooting parameters of the captured rail image.

[0038] In another embodiment, the rail structure parameter can include at least one of the following: pixel number corresponding to the length and width of the rail in the rail image, pixel number corresponding to the size parameter of the fastener in the rail image, pixel number corresponding to the size parameter of the tie in the rail image, pixel number corresponding to the distance of the tie relative to the rail in the rail image, pixel number corresponding to the position of the fastener relative to the tie in the rail image, pixel number corresponding to the position of the fastener relative to the rail in the rail image, etc. ​

[0039] In step S102, a position coordinate of the track in the horizontal direction in the target track image is determined, wherein the horizontal direction is a direction perpendicular to the extension direction of the track in the target track image.

[0040] After the target track image is obtained, the position coordinate of the track in the horizontal direction in the target track image can be determined. The horizontal direction refers to a direction perpendicular to the extension direction of the track in the target track image. In an embodiment, the position coordinate of the track in the horizontal direction can be determined according to the shooting parameters of the camera shooting the target track image. For example, the position of the camera relative to the track and the camera shooting parameters can be fixed when shooting, and then the position coordinate of the track in the horizontal direction in the target track image can be determined according to the relative position and the shooting parameters.

[0041] In step S103, a detection region is selected from the target track image based on the position coordinate of the track in the horizontal direction in the target track image and the track structure parameters, wherein the detection region includes a fastener region.

[0042] After the position coordinate of the track in the horizontal direction in the target track image is determined, the detection region can be selected from the target track image based on the position coordinate and the track structure parameters. The detection region can include the fastener region. In an embodiment, the position parameter of the fastener relative to the track in the track image can be determined according to the positional relationship of the fastener relative to the track and the structure parameters of the fastener and the camera shooting parameters. The general region of the fastener in the track image can be determined based on the position coordinate of the track in the horizontal direction in the target track image and the position parameter of the fastener relative to the track in the track image. In another embodiment, the general region of the fastener in the track image, i.e., the fastener region, can be determined according to the number of corresponding pixels of the size parameter of the fastener in the track image and the number of corresponding pixels of the position of the fastener relative to the track in the track image.

[0043] In another embodiment, the detection region can include the fastener region and the sleeper region. The general region of the sleeper in the track image can be determined based on the position coordinate of the track and the positional relationship of the sleeper relative to the track and the structure parameters of the sleeper. The general region of the sleeper in the track image, i.e., the sleeper region, can also be determined according to the number of corresponding pixels of the size parameter of the sleeper in the track image and the number of corresponding pixels of the position of the sleeper relative to the track in the track image.

[0044] In step S104, a candidate region is extracted from the detection region, wherein the candidate region includes a fastener candidate rectangular frame.

[0045] After the detection region is selected, a candidate region can be extracted from the detection region. Specifically, the candidate region can include a fastener candidate rectangular frame. Specifically, the coordinates of each fastener in the vertical direction can be determined first, and then the candidate rectangular frame can be determined in combination with the coordinates of each fastener in the vertical direction and the detection region, to obtain the candidate region. The candidate region can include multiple candidate rectangular frames.

[0046] In step S105, the candidate region is classified based on the neural network to identify the fastener from the candidate region.

[0047] After the candidate region is extracted, the candidate region can be classified based on the neural network to identify the fastener from the candidate region, to determine whether the fastener exists in the candidate region, thereby performing fastener identification and positioning.

[0048] In the method in the above embodiments, after the target track image is obtained, the position of the track in the target track image can be determined, and according to the position of the track and the track structure parameters, a detection region including a fastener region can be selected, and a candidate region including a fastener candidate rectangular frame can be extracted from the detection region. By selecting the detection region in combination with the prior knowledge of the track structure parameters, the efficiency and accuracy of the detection region picking can be improved, and the amount of calculation can be reduced. Then, the fastener can be identified from the candidate region based on the neural network, which can further improve the identification accuracy of the fastener, and determine whether the fastener exists in the candidate region. Not only can the track fastener positioning be accurately implemented, but also the real-time requirement of the high-speed railway for fastener positioning and identification can be guaranteed.

[0049] In some embodiments of the present specification, determining the position coordinates of the track in the target track image in the horizontal direction can include: performing gray projection on the target track image along the vertical direction to determine the position coordinates of the track in the target track image in the horizontal direction, wherein the vertical direction is the extension direction of the track in the target track image.

[0050] A direct longitudinal gray projection method can be used. The original image data is projected in the vertical direction to obtain a gray projection histogram of the entire image. In an embodiment, 95% of the maximum value in the projection histogram (the percentage can be selected according to actual needs, for example, the value range is 85%-98%) can be taken as a histogram statistical threshold:

[0051] track_threshold = max_bin_value x 0.95;

[0052] Wherein, track_threshold is the statistical threshold, and max_bin_value is the maximum value in the projection histogram.

[0053] After obtaining the gray scale projection histogram, the histogram can be traversed along the y direction (i.e., the vertical direction), the longest continuous interval greater than the threshold value is taken, and the midpoint is taken as the position coordinate x_track of the track in the x direction (i.e., the horizontal direction). After obtaining the position coordinate of the track in the horizontal direction, the detection region can be selected according to the position coordinate and the track structure parameter. In the above manner, the position coordinate of the track in the horizontal direction can be determined, and then the detection region is selected.

[0054] In some embodiments of the present specification, extracting the candidate region from the detection region can include: performing region binarization processing on the detection region to obtain a binarization-processed detection region; and performing gray scale projection on the binarization-processed detection region along the horizontal direction to extract the candidate region from the detection region.

[0055] Specifically, different threshold values can be used for region-adaptive binarization according to different light and dark levels of the detection region (fastener region + sleeper region) image, so as to reduce the influence of light on fastener positioning. The fastener region and the sleeper region can be combined into a detection region, and the threshold values of the original image can be calculated regionally. The threshold value calculation method is to sort all pixels in the region from low to high according to the gray scale value, and the gray scale value corresponding to the top 10% (which can be set according to actual needs, for example, 5%-15%) of points in the fastener region is taken as the threshold value. The gray scale value corresponding to the top 20% (which can be set according to actual needs, for example, 15%-25%) of points in the detection region is taken as the threshold value, and the detection region is binarized with the threshold value of the detection region. When the threshold value of the fastener region is greater than the threshold value of the detection region, the fastener region is binarized separately with the threshold value of the fastener region, and finally the region binarization algorithm is completed. The horizontal direction gray scale projection histogram of the image fastener region and the sleeper region is calculated for accurate positioning of the fastener. The fastener detection threshold value is set at 90% (which can be set according to actual needs, for example, 85%-95%) of the width of the fastener region, the horizontal histogram is counted along the horizontal direction in the fastener detection region, and the coordinates of the fastener along the vertical direction are calculated. In an embodiment, each value of the horizontal direction gray scale projection histogram can be traversed, and if there are two consecutive values greater than the fastener detection threshold value, the y coordinate corresponding to the point is the top point coordinate of the fastener. When the value of the projection histogram is less than the fastener detection threshold value, the y coordinate corresponding to the point is the bottom point coordinate of the fastener, and the horizontal coordinate of the fastener region forms a detection fastener candidate rectangular frame. In the above manner, the candidate region can be extracted from the detection region.

[0056] In some embodiments of the present specification, classifying the candidate region based on a neural network to identify the fastener from the candidate region can include: classifying the candidate region using a ResNet18 neural network to determine whether the fastener exists in the candidate region.

[0057] To improve the efficiency of fastener recognition, a lightweight neural network can be used to classify the candidate region. In this embodiment, a ResNet18 neural network can be used to classify the candidate region to determine whether a fastener exists in the candidate region. Of course, other lightweight neural networks can also be used to classify the candidate region to determine whether a fastener exists in the region in other embodiments of the present specification. In this way, the efficiency of fastener recognition and positioning can be improved.

[0058] Further, in some embodiments of the present specification, the activation function in the ResNet18 neural network can be a Mish activation function. In this embodiment, the Mish activation function can be used to replace the original activation function Relu activation function in the convolutional layer of the ResNet18 neural network. The Mish activation function is smoother than the Relu activation function, has no non-differentiable point globally, and is similar to the Relu function in the positive interval, but still has a gradient in the negative interval. Only when it tends to -∞, there is a gradient disappearance situation, which is beneficial to the derivative operation in the back propagation process, and can obtain better feature transmission ability and generalization ability. In this way, the accuracy of fastener recognition can be improved.

[0059] In some embodiments of the present specification, two CBM modules and an identity mapping channel fusion parallel structure can be used in the residual module of the ResNet18 neural network.

[0060] Specifically, the residual module of the ResNet18 neural network can be improved, and the improved residual module uses a parallel structure of two CBM modules and an identity mapping channel fusion. First, the input feature data is copied into two equal parts, part1 and part2. Then, part2 is processed through two CBM modules to obtain processed part2, and part1 and processed part2 are added to obtain the final feature data. Compared with the previous residual network, this residual network structure greatly reduces the computational complexity and improves the feature extraction ability of the network, which can meet the real-time and high-precision requirements of fastener recognition and classification.

[0061] In some embodiments of the present specification, after extracting the candidate region from the detection region, the method can further include: in a case where the number of fastener candidate rectangular frames in the candidate region does not satisfy a preset condition, obtaining a second track image adjacent to the target track image; splicing a preset part of the target track image and the second track image to obtain a spliced target image, and extracting a candidate region from the spliced target image.

[0062] In the process of train driving, when the camera shoots multiple track images, the multiple track images are spliced to form a photo of the whole track. In the target track image, there may be incomplete fasteners, in which case, splicing with adjacent track images is needed to avoid missing fasteners. The number of fasteners in the normally shot track image is fixed. In the case that the number of fastener candidate rectangular frames in the candidate region in the target track image is less than the preset number, a second track image is acquired. The second track image is the next track image adjacent to the target track image. The target track image and the preset part of the second track image can be spliced to obtain a spliced target image, and the candidate region is extracted from the spliced target image. The specific extraction method is as described above. The preset part of the second track image can be the end of the second track image adjacent to the target track image, and the specific length of the cut is not greater than the length of the fastener along the vertical direction. The splicing direction is along the direction of the track extension. Through the above method, the missing of fasteners in the fastener detection process can be avoided, and the accuracy of fastener detection can be further improved.

[0063] Each of the embodiments in the specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the difference from other embodiments. For specific description, refer to the description of the related processing embodiments described above, which will not be repeated here.

[0064] The above describes specific embodiments of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different than the order in the embodiments and still achieve the desired result. In addition, the processes depicted in the figures do not necessarily require the particular order shown or sequential order to achieve the desired results. In some embodiments, multi-task processing and parallel processing are possible or advantageous.

[0065] The above method is described in combination with a specific embodiment, however, it is worth noting that the specific embodiment is only for better illustrating the specification and does not constitute an improper limitation on the specification.

[0066] In a specific embodiment of the present invention, a traditional image positioning algorithm based on prior information is combined with a deep learning recognition algorithm based on a convolutional neural network to quickly locate fasteners from track images and identify their status. The fast fastener positioning and recognition method proposed in this embodiment can locate fastener positions faster than general deep learning algorithms because it introduces prior knowledge of the track structure; and because a deep learning network is used to identify the fastener status, it can identify the fastener status more accurately than traditional methods. Because the algorithm in this specific embodiment uses a traditional algorithm based on prior information and a lightweight convolutional neural network, it has a fast computing speed and occupies little hardware resources. It can be deployed in hardware environments with limited computing speed and hardware resources, such as embedded devices.

[0067] Please refer to Figure 3 , shows a flow chart of a method for identifying fasteners in track images based on structural priors in an embodiment of this specification. Specifically, Figure 3 As shown, the detection method proposed in this specific embodiment can be divided into two parts, which are performed on the CPU side and the GPU side respectively: (1) An image processing method is used on the CPU side to extract candidate fastener areas, which specifically includes five parts: track position detection, detection area selection, area binarization, horizontal histogram calculation and candidate area extraction; (2) A deep learning method is used on the GPU side to classify fasteners, and the specific algorithm uses a simple ResNet18 neural network. The original track image to be processed in this specific embodiment is as follows Figure 2 shown.

[0068] like Figure 3 As shown, the track position is detected first. Track position detection uses the method of direct longitudinal grayscale projection. The original image data is grayscale projected along the vertical direction to obtain the grayscale projection histogram of the entire image. The 95% of the maximum value in the projection histogram is found as the histogram statistical threshold:

[0069] track_threshold=max_bin_value×0.95;

[0070] Among them, track_threshold is the statistical threshold, and max_bin_value is the maximum value in the projection histogram.

[0071] After obtaining the grayscale projection histogram, the histogram can be traversed along the y direction (ie, vertical direction), the longest continuous interval greater than the threshold is taken, and the midpoint is taken as the position coordinate x_track of the track in the x direction (ie, horizontal direction).

[0072] Next, the detection area is selected. Based on the calculated track direction coordinates, the coordinate range of the fastener areas on both sides is obtained according to prior knowledge:

[0073] [x_track-120-140, x_track-120], [x_track+100, x_track+100+140].

[0074] The sleeper reference region is:

[0075] [x_track-120-140-300, x_track-120-140], [x_track+100+140, x_track+100+140+300].

[0076] Wherein, 120 is the number of pixels between the left track and the fastener, 100 is the number of pixels between the right track and the fastener, 140 is the width of the fastener region, and 300 is the width of the sleeper region. The selection of prior knowledge is related to the setting of the collection camera. In this specific embodiment, the physical resolution of the line array camera is 1 mm / pix. Please refer to Figure 4 , which shows the detection region identification diagram of the track image in an embodiment of the present specification. As shown in Figure 4 , 41 is the sleeper region, and 42 is the fastener region.

[0077] After obtaining the detection region, the detection region can be subjected to region binarization processing. According to the different light and dark levels of the detection region (fastener region + sleeper region) image, a region adaptive binarization method with different thresholds can be used to reduce the influence of light on fastener positioning. Please refer to Figure 5 , which shows the region binarization algorithm result of the detection region of the track image in an embodiment of the present specification. Please refer to Figure 6 , which shows the algorithm flowchart of the region binarization of the detection region of the track image in an embodiment of the present specification. The fastener region and the sleeper region can be combined into a detection region, and the threshold values of the original image are calculated regionally. The threshold calculation method is to sort all pixels in the region from low to high according to the gray value. The gray value corresponding to the first 10% of points in the fastener region is taken as the threshold value, the gray value corresponding to the first 20% of points in the detection region is taken as the threshold value, and the detection region is subjected to binarization processing with the threshold value of the detection region. When the threshold value of the fastener region is greater than the threshold value of the detection region, the fastener region is subjected to binarization processing with the threshold value of the fastener region alone, and the region binarization algorithm is finally completed.

[0078] Horizontal histogram calculation can be performed to calculate the horizontal direction gray projection histogram of the image fastener region and the sleeper region, respectively, for precise fastener positioning.

[0079] After that, candidate region extraction is performed. Specifically, a fastener detection threshold can be set at 90% of the width of the fastener region, and a horizontal histogram is counted in the x direction within the fastener detection region to calculate the y coordinate of the fastener. The specific method is as follows: traverse each value of the horizontal gray projection histogram, if there are two consecutive values greater than the fastener detection threshold, then the y coordinate corresponding to the point is the top point coordinate top y of the fastener, and until the value of the projection histogram is less than the fastener detection threshold, the y coordinate corresponding to the point is the bottom point coordinate bottom y of the fastener, and the horizontal coordinate of the fastener region is combined to form a detection fastener candidate rectangular frame.

[0080] After the candidate region is extracted, the candidate region can be classified based on deep learning on the GPU side. The original residual network structure expression of the neural network is as follows:

[0081] y = H (x, W H ) + x

[0082] When an input is directly added to the output, the output y can be explicitly split into H (x, Wh) and the linear superposition of the input x, so that the gradient has an identity mapping channel, which is considered to be very important for the training of deep networks. Then, the horizontal gray projection histogram of the image fastener region and the sleeper region can be calculated respectively for precise fastener positioning.

[0083] When extracting the candidate region, a fastener detection threshold can be set at 90% of the width of the fastener region, and a horizontal histogram is counted in the x direction within the fastener detection region to calculate the y coordinate of the fastener. The specific method is as follows: traverse each value of the horizontal gray projection histogram, if there are two consecutive values greater than the fastener detection threshold, then the y coordinate corresponding to the point is the top point coordinate top y of the fastener, and until the value of the projection histogram is less than the fastener detection threshold, the y coordinate corresponding to the point is the bottom point coordinate bottom y of the fastener, and the horizontal coordinate of the fastener region is combined to form a detection fastener candidate rectangular frame.

[0084] After obtaining the candidate region including multiple fastener candidate rectangular frames, the candidate region can be classified based on deep learning (on the GPU side). The Mish activation function can be used as the activation function of ResNet18. Please refer to Figure 7 , which shows the curve diagram of the Mish activation function. The difference between the Mish activation function and the Relu activation function lies in its derivative. The derivative of the Relu activation function is as follows,

[0085]

[0086] The derivative expression of the Mish activation function is as follows:

[0087]

[0088] As can be seen from the above two formulas, the Relu activation function is not derivable at x = 0, and the derivative is always 0 in the interval x < 0, which may cause the gradient to disappear when the convolutional neural network is reversed to calculate the gradient. The Mish activation function is derivable globally, and the derivative can still be calculated in the interval x < 0; when x tends to infinity, the derivative tends to 0. Therefore, the expression ability of the Mish activation function is better than that of the Relu activation function.

[0089] In the specific embodiment, the idea of Yolo v4 can also be used to improve the residual module to adopt a parallel structure of two CBM modules and an identity mapping channel, as shown in Figure 8 First, the input feature data (Input) is copied into two equal parts (Part1 and Part2). Then, Part2 is processed through two CBM modules to obtain processed Part2, and Part1 and processed Part2 are added (Add) to obtain the final feature data. As shown in Figure 8 The CBM module is Conv (convolution) plus BN (Batch Normalization) plus Mish function. Compared with the previous residual network, this residual network structure greatly reduces the computational complexity and improves the feature extraction capability of the network, which can meet the real-time and high-precision requirements of fastener recognition and classification.

[0090] The fastener candidate region can be cropped into an image with a resolution size of 224x224. The rectangular candidate region is filled with 0 values, and the residual structure is used to extract features. Through the softmax classifier, the candidate region is divided into two categories: fastener and non-fastener. The interference region in the candidate region is removed, and finally the accurate detection result of the fastener region is obtained. Please refer to Figure 9 , which shows a schematic diagram of the fastener detection and classification result in an embodiment of the present specification.

[0091] The scheme in the above embodiment proposes a fast detection algorithm for rail fasteners. In the original two-step detection of FAST-RCNN, the Selective Search is replaced by a gray-scale projection method with stronger pertinence and lower computational complexity. In the classification and positioning stage, a simpler ResNet18 is used, and the fastener position regression part is abandoned. Without affecting the recognition accuracy, the model has fewer parameters and is easier to train. In the above scheme, an improved ResNet18 network is also proposed. The original Relu activation function is replaced by a Mish activation function, and the residual module structure is replaced by two stacked CBM modules. This not only greatly reduces the computational complexity, but also improves the feature extraction capability of the network, which can meet the real-time and high-precision requirements of fastener identification and classification. In addition, the candidate region algorithm uses CPU calculation, and the deep learning classification uses GPU operation. The two steps can be processed in a pipeline parallel manner, which can fully utilize the computing resources and is suitable for embedded detection platforms.

[0092] Based on the same inventive concept, the embodiments of the present specification also provide a rail image fastener identification device based on structural prior, which is an embedded device and includes a CPU end and a GPU end. The CPU end is configured to: obtain a target rail image and a rail structure parameter; determine a horizontal position coordinate of a rail in the target rail image; select a detection region from the target rail image based on the horizontal position coordinate of the rail in the target rail image and the rail structure parameter; and extract a candidate region from the detection region; wherein the detection region includes a fastener region, and the candidate region includes a fastener candidate rectangular frame. The GPU end is configured to: classify the candidate region based on a neural network to identify a fastener from the candidate region.

[0093] Based on the same inventive concept, the embodiments of the present specification also provide a rail image fastener identification device based on structural prior, which is an embedded device and includes a CPU end and a GPU end. The CPU end is configured to: obtain a target rail image and a rail structure parameter; determine a horizontal position coordinate of a rail in the target rail image; select a detection region from the target rail image based on the horizontal position coordinate of the rail in the target rail image and the rail structure parameter; and extract a candidate region from the detection region; wherein the detection region includes a fastener region, and the candidate region includes a fastener candidate rectangular frame. The GPU end is configured to: classify the candidate region based on a neural network to identify a fastener from the candidate region. Figure 10 is a structural block diagram of the rail image fastener identification device based on structural prior of the embodiments of the present specification, as Figure 10As shown, the device includes an acquisition module 1001, a determination module 1002, a selection module 1003, an extraction module 1004, and an identification module 1005, which are described below.

[0094] The acquisition module 1001 is configured to acquire a target track image and track structure parameters.

[0095] The determination module 1002 is configured to determine a horizontal position coordinate of a track in the target track image, wherein the horizontal direction is a direction perpendicular to the extension direction of the track in the target track image.

[0096] The selection module 1003 is configured to select a detection region from the target track image based on the horizontal position coordinate of the track in the target track image and the track structure parameters, wherein the detection region includes a fastener region.

[0097] The extraction module 1004 is configured to extract a candidate region from the detection region, wherein the candidate region includes a fastener candidate rectangular frame.

[0098] The identification module 1005 is configured to classify the candidate region based on a neural network to identify a fastener from the candidate region.

[0099] In some embodiments of the present disclosure, the determination module can be specifically configured to perform gray projection on the target track image along a vertical direction to determine the horizontal position coordinate of the track in the target track image, wherein the vertical direction is the extension direction of the track in the target track image.

[0100] In some embodiments of the present disclosure, the extraction module can be specifically configured to perform region binarization processing on the detection region to obtain a binarization-processed detection region, and perform gray projection on the binarization-processed detection region along a horizontal direction to extract a candidate region from the detection region.

[0101] In some embodiments of the present disclosure, the identification module can be specifically configured to classify the candidate region using a ResNet18 neural network to determine whether a fastener exists in the candidate region.

[0102] In some embodiments of the present disclosure, the activation function in the ResNet18 neural network can be a Mish activation function.

[0103] In some embodiments of the present disclosure, a parallel structure of two CBM modules and an identity mapping channel can be used in the residual module of the ResNet18 neural network.

[0104] In some embodiments of the present specification, the extraction module can be further specifically configured to: after extracting the candidate region from the detection region, in a case where the number of fastener candidate rectangular frames in the candidate region does not satisfy a preset condition, acquire a second track image, wherein the second track image is adjacent to the target track image; splice the target track image and a preset part of the second track image to obtain a spliced target image, and extract a candidate region from the spliced target image.

[0105] From the above description, it can be seen that the embodiments of the present specification achieve the following technical effects: after obtaining the target track image, the position of the track in the target track image can be determined, and according to the position of the track and the track structure parameters, a detection region including a fastener region can be selected, and a candidate region containing a fastener candidate rectangular frame can be extracted from the detection region. By combining the prior knowledge of the track structure parameters to select the detection region, the efficiency and accuracy of the detection region picking can be improved, and the amount of calculation can be reduced. Then, the fastener is identified from the candidate region based on the neural network, which can further improve the identification accuracy of the fastener, and determine whether the fastener exists in the candidate region. Not only can the track fastener positioning be accurately realized, but also the real-time requirement of the high-speed railway for fastener positioning and identification can be guaranteed. The above-mentioned scheme solves the problem that the fastener detection method in the prior art cannot meet the real-time requirement, and achieves the technical effect of effectively improving the fastener identification efficiency and accuracy while ensuring the identification accuracy.

[0106] The present specification also provides a computer device, which can specifically refer to Figure 11 The computer device for implementing the fastener identification method based on structure prior of the track image provided by the embodiments of the present specification can specifically include an input device 111, a processor 112, and a memory 113. The memory 113 is used to store processor executable instructions. The processor 112 executes the instructions to implement the steps of the fastener identification method based on structure prior of the track image in any of the above embodiments.

[0107] In the embodiment, the input device can be specifically one of main devices for information exchange between the user and the computer system. The input device can include a keyboard, a mouse, a camera, a scanner, a light pen, a handwriting input board, a voice input device, etc.; the input device is used to input raw data and programs for processing the data into the computer. The input device can also acquire data transmitted by other modules, units, devices. The processor can be implemented in any appropriate manner. For example, the processor can take the form of a microprocessor or a processor and a computer readable medium storing computer readable program code (such as software or firmware) executable by the (micro)processor, logic gates, switches, application specific integrated circuits (ASIC), programmable logic controllers, and embedded microcontrollers, etc. The memory can be specifically a memory device for saving information in modern information technology. The memory can include multiple levels, and in a digital system, as long as it can save binary data, it can be a memory; in an integrated circuit, a circuit without a physical form and with a storage function is also called a memory, such as RAM, FIFO, etc.; in a system, a storage device with a physical form is also called a memory, such as a memory stick, a TF card, etc.

[0108] In the embodiment, the functions and effects realized by the computer device can be explained in comparison with other embodiments, and will not be repeated here.

[0109] The present specification also provides a computer storage medium for a track image fastener recognition method based on structural priori, which stores computer program instructions. When the computer program instructions are executed, the steps of the track image fastener recognition method based on structural priori in any of the above embodiments are realized.

[0110] In the embodiment, the storage medium includes but is not limited to a random access memory (RAM), a read-only memory (ROM), a cache, a hard disk drive (HDD), or a memory card. The memory can be used to store computer program instructions. The network communication unit can be an interface set according to the standard of a communication protocol, used for network connection communication.

[0111] In the embodiment, the functions and effects realized by the program instructions stored in the computer storage medium can be explained in comparison with other embodiments, and will not be repeated here.

[0112] Obviously, those skilled in the art should understand that each module or each step of the above-mentioned embodiments of the present description can be realized by a general computing device, which can be centralized on a single computing device or distributed on a network composed of multiple computing devices, and optionally, each module or each step can be realized by program codes executable by a computing device, so that each module or each step can be stored in a storage device and executed by a computing device, and in some cases, the steps shown or described can be executed in different order, or each module or step can be manufactured into an individual integrated circuit module, or multiple modules or steps can be manufactured into a single integrated circuit module. Thus, the embodiments of the present description are not limited to any specific combination of hardware and software.

[0113] It is to be understood that the above description is intended to be illustrative and not restrictive. Many embodiments and applications other than the examples provided would be apparent to those of skill in the art upon reading the above description. The scope of the description should be determined, not with reference to the above description, but should instead be determined with reference to the appended claims, along with the full range of equivalents to which such claims are entitled. It is intended that the description cover all such modifications, equivalents, and alternatives. This description is intended to be illustrative, and not to limit the scope of the claims. What is claimed is:

[0114] The above description is only the preferred embodiments of the present description and is not intended to limit the present description. The embodiments of the present description can have various modifications and changes for those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present description shall be included in the protection scope of the present description.

Claims

1. A method for rail image fastener recognition based on structural prior, characterized in that, The method comprises the following steps: acquiring a target track image and track structure parameters; determining a horizontal position coordinate of a track in the target track image, wherein the horizontal direction is perpendicular to the extension direction of the track in the target track image; selecting a detection region from the target track image based on the horizontal position coordinate of the track in the target track image and the track structure parameters, wherein the detection region comprises a fastener region; extracting a candidate region from the detection region, wherein the candidate region comprises a fastener candidate rectangular frame; classifying the candidate region based on a neural network to identify a fastener from the candidate region; wherein the detection region comprises a fastener region and a sleeper region; accordingly, extracting a candidate region from the detection region comprises: performing a binaryzation processing on the detection region in a region-adaptive binaryzation manner based on different image brightness levels of the fastener region and the sleeper region to obtain a binaryzation-processed detection region; specifically, sorting all pixels in the detection region from low to high according to the gray value, taking the gray value corresponding to the first preset percentage of points in the fastener region as the first threshold value corresponding to the fastener region, taking the gray value corresponding to the second preset percentage of points in the detection region as the second threshold value corresponding to the detection region, performing binaryzation processing on the detection region by using the second threshold value, and performing binaryzation processing on the fastener region by using the first threshold value alone in the case that the first threshold value is greater than the second threshold value; the first preset percentage is 5% to 15%, and the second preset percentage is 15% to 25%; performing gray projection on the binaryzation-processed detection region along the horizontal direction to extract a candidate region from the detection region; specifically, calculating the horizontal direction gray projection histogram of the fastener region and the sleeper region respectively, setting a fastener detection threshold value at the third preset percentage of the fastener region width, counting the horizontal histogram along the horizontal direction in the fastener detection region, calculating the coordinates of the fastener along the vertical direction, traversing each value of the horizontal direction gray projection histogram, if there are two consecutive values greater than the fastener detection threshold value, the vertical coordinate corresponding to the point is the top point coordinate of the fastener, and until the value of the projection histogram is less than the fastener detection threshold value, the vertical coordinate corresponding to the point is the bottom point coordinate of the fastener, combining the horizontal coordinate of the fastener region to form a detection fastener candidate rectangular frame, and the third preset percentage is 85% to 95%.

2. The structure prior based track image fastener recognition method according to claim 1, characterized in that, determining the horizontal position coordinate of the track in the target track image comprises: performing gray projection on the target track image along the vertical direction to determine the horizontal position coordinate of the track in the target track image, wherein the vertical direction is the extension direction of the track in the target track image.

3. The structural prior based track image fastener recognition method of claim 1, wherein, classifying the candidate region based on a neural network to identify a fastener from the candidate region comprises: The ResNet18 neural network is used to classify the candidate region to determine whether the fastener exists in the candidate region.

4. The structural prior based track image fastener recognition method of claim 3, wherein, The activation function in the ResNet18 neural network is a Mish activation function.

5. The structural prior based track image fastener identification method of claim 3, wherein, The parallel structure of two CBM modules and an identity mapping channel is used in the residual module of the ResNet18 neural network.

6. The structural prior based track image fastener identification method of claim 1, wherein, After the candidate region is extracted from the detection region, further comprising: In a case where the number of fastener candidate rectangular frames in the candidate region does not satisfy a preset condition, a second track image adjacent to the target track image is acquired; The target track image and a preset part of the second track image are spliced to obtain a spliced target image, and a candidate region is extracted from the spliced target image.

7. A structure-prior-based rail image fastening recognition device, characterized by, The track image fastener recognition device is an embedded device, including a CPU end and a GPU end. The CPU end is configured to acquire a target track image and track structure parameters, determine the position coordinates of a track in the target track image in the horizontal direction, select a detection region from the target track image based on the position coordinates of the track in the target track image in the horizontal direction and the track structure parameters, and extract a candidate region from the detection region; wherein the detection region includes a fastener region, and the candidate region includes a fastener candidate rectangular frame. The GPU end is configured to classify the candidate region based on a neural network to identify a fastener from the candidate region. The detection region includes a fastener region and a sleeper region. Correspondingly, the CPU end is configured to: according to different image brightness levels corresponding to the fastener region and the sleeper region, adopt a different threshold value region-adaptive binarization manner to perform binarization processing on the detection region to obtain a binarization-processed detection region; specifically, all pixels in the detection region are sorted from low to high according to gray values, a gray value corresponding to a first preset percentage of points in the fastener region is taken as a first threshold value corresponding to the fastener region, a gray value corresponding to a second preset percentage of points in the detection region is taken as a second threshold value corresponding to the detection region, the second threshold value is used to perform binarization processing on the detection region, and in the case that the first threshold value is greater than the second threshold value, the first threshold value is used to perform binarization processing on the fastener region alone; the first preset percentage is 5% to 15%, and the second preset percentage is 15% to 25%; a gray projection is performed on the binarization-processed detection region along a horizontal direction to extract a candidate region from the detection region; specifically, a horizontal direction gray projection histogram of the fastener region and the sleeper region is calculated respectively, a fastener detection threshold value is set at a third preset percentage of the width of the fastener region, a horizontal histogram is counted along the horizontal direction in the fastener detection region, a coordinate of the fastener along the vertical direction is calculated, each value of the horizontal direction gray projection histogram is traversed, if there are two consecutive values greater than the fastener detection threshold value, a vertical coordinate corresponding to the point is the top point coordinate of the fastener, and until the value of the projection histogram is less than the fastener detection threshold value, a vertical coordinate corresponding to the point is the bottom point coordinate of the fastener, and a detection fastener candidate rectangular frame is formed in combination with the horizontal coordinate of the fastener region; the third preset percentage is 85% to 95%.

8. A structure-prior-based rail image fastening recognition device, characterized by, The method comprises: an acquisition module configured to acquire a target track image and track structure parameters; a determination module configured to determine a horizontal position coordinate of a track in the target track image, wherein the horizontal direction is a direction perpendicular to an extension direction of the track in the target track image; a selection module configured to select a detection region from the target track image based on the horizontal position coordinate of the track in the target track image and the track structure parameters, wherein the detection region includes a fastener region; an extraction module configured to extract a candidate region from the detection region, wherein the candidate region includes a fastener candidate rectangular frame; an identification module configured to classify the candidate region based on a neural network to identify a fastener from the candidate region. The detection region includes a fastener region and a sleeper region. Correspondingly, the extraction module is specifically configured to: according to different image brightness levels corresponding to the fastener region and the sleeper region, adopt a different threshold value region-adaptive binarization manner to perform binarization processing on the detection region to obtain a binarization-processed detection region; specifically, all pixels in the detection region are sorted from low to high according to the gray value, the gray value corresponding to the first preset percentage of points in the fastener region is taken as the first threshold value corresponding to the fastener region, the gray value corresponding to the second preset percentage of points in the detection region is taken as the second threshold value corresponding to the detection region, the second threshold value is used to perform binarization processing on the detection region, and in the case that the first threshold value is greater than the second threshold value, the first threshold value is used to perform binarization processing on the fastener region alone; the first preset percentage is 5% to 15%, and the second preset percentage is 15% to 25%; the binarization-processed detection region is projected in the horizontal direction to extract a candidate region from the detection region; specifically, the horizontal direction gray value projection histogram of the fastener region and the sleeper region is calculated respectively, the third preset percentage of the fastener region width is taken as a fastener detection threshold value, the horizontal histogram is counted in the fastener detection region along the horizontal direction, the coordinates of the fastener along the vertical direction are calculated, each value of the horizontal direction gray value projection histogram is traversed, if there are two continuous values greater than the fastener detection threshold value, the vertical coordinate corresponding to the point is the top point coordinate of the fastener, until the value of the projection histogram is less than the fastener detection threshold value, the vertical coordinate corresponding to the point is the bottom point coordinate of the fastener, and the horizontal coordinate of the fastener region is combined to form a detection fastener candidate rectangular frame; the third preset percentage is 85% to 95%.

9. A computer device, comprising: The computer program product comprises a processor and a memory for storing processor-executable instructions, and the processor executes the instructions to implement the steps of the method in any one of claims 1 to 6.

Images