Track gap anomaly detection method, device, terminal and medium based on deep learning

Abstract

The present invention provides a track gap anomaly detection method, device, terminal and medium based on deep learning; wherein the method comprises: using a trained machine learning model to perform target gap recognition on a current optical image; when determining that the current optical image contains a target gap, extracting width information of the target gap from the current optical image and extracting depth information of the target gap from the current depth image according to the recognition range of the target gap, so as to perform target gap anomaly detection based on the width information and depth information of the target gap; the present invention can quickly and accurately obtain the target gap anomaly detection result corresponding to each acquisition moment.

Description

Technical Field

[0001] The present invention relates to the technical field of nondestructive testing, and in particular to a method, device, terminal and computer storage medium for detecting track surface gaps based on deep learning. Background Art

[0002] After the track is laid, joints will be formed between the tracks, and expansion joints will be set at intervals between the tracks to adapt to the characteristics of the track's thermal expansion and contraction deformation. Also, during the track operation, damaged gaps will appear on the track surface due to wear. Therefore, there will be a large number of gaps on the track or between the tracks. In order to improve the operational safety of rail transit, it is usually necessary to detect the damaged gaps on the track surface caused by wear to avoid increasing the operational risk of the track due to the excessive width of the damaged gaps.

[0003] In the existing track gap detection process, the detection personnel usually use a scanning device to collect a scanned image of the track, and extract the damaged gap and determine whether the damaged gap is abnormal by extracting the gap information in the scanned image. This detection method is not only inefficient, but also requires a lot of manpower. In addition, since the width of the gap is usually small, at the centimeter level, the extracted gap information is often less accurate. In addition, since there are other normal gaps such as splicing joints and expansion joints in the track in addition to the damaged gap, it is easy to cause misjudgment between the damaged gap and other gaps, thereby further reducing the accuracy and efficiency of abnormal detection of damaged gaps in the track. Summary of the invention

[0004] In view of the shortcomings of the prior art mentioned above, the purpose of the present invention is to provide a method, device, terminal and medium for detecting track surface gaps based on deep learning, so as to solve the problems that the gap width collected by the existing track gap detection method has low accuracy and poor extraction efficiency.

[0005] To achieve the above-mentioned purpose and other related purposes, the present invention provides, in a first aspect, a track gap anomaly detection method based on deep learning. During the track inspection process, based on the image pairs collected at each collection moment, the target gap anomaly detection process is repeatedly performed; wherein, the image pairs include depth images and optical images; for a single collection moment, the track gap anomaly detection method based on deep learning includes: using a trained machine learning model to perform target gap recognition on the current optical image; when the current optical image is determined to contain a target gap, executing the following steps: extracting the recognition range of the target gap in the current optical image; extracting the width information of the target gap in the current optical image according to the recognition range of the target gap, and extracting the depth information of the target gap in the current depth image according to the recognition range of the target gap; performing target gap anomaly detection based on the width information and depth information of the target gap to obtain anomaly detection results corresponding to each collection moment.

[0006] In some embodiments of the first aspect of the present invention, the loss function of the machine learning model is a function constructed based on the classification category loss function and the recognition range loss function of the target gap sample, which is:

[0007] in,

[0008] a is the weight coefficient; N is the number of samples in the training set; L classifer is the classification category loss function of the target gap sample; L iou is the recognition range loss function of the target gap sample; L i For the i The loss function of target gap samples; y i For the i The identification value of the target gap sample; p i For the i The probability value of the target gap sample; is the learning rate; v is the learning step length; b and b gt Respectively represent the position of the recognition range and the position of the corresponding identification range of the target gap sample; c It is the distance between the recognition range and the identification range corresponding to the target gap sample.

[0009] In some embodiments of the first aspect of the present invention, the method for extracting the width information of the target gap in the current optical image based on the recognition range of the target gap includes: based on the recognition range, extracting the target segment of the target gap within the recognition range in the optical image; within the target segment, extracting the vertical distance value of each pixel point on the two side lines of the target gap, so as to obtain the width information of the target gap based on each distance value; wherein the width information is used to characterize the distance feature between the two side lines of the target gap.

[0010] In some embodiments of the first aspect of the present invention, when the target gap is an irregular gap, the method for extracting the vertical distance value of each pixel point on the two side lines of the target gap in the target segment includes: in the target segment, performing linear fitting on the first side line and the second side line of the target gap respectively to obtain corresponding fitting lines; determining the vertical direction of each fitting line, and extracting the distance value between each pixel point on the fitting line and the other side line along the vertical direction.

[0011] In some embodiments of the first aspect of the present invention, when the target gap is an expansion joint, the recognition range of the target gap is a local recognition frame covering a local area of ​​the expansion joint; wherein the local recognition frame is used to identify an enclosed area enclosed by the table tops in the end surfaces on both sides of the expansion joint; the implementation method of extracting the width information of the target gap in the current optical image based on the recognition range of the target gap includes: extracting the distance between the first side line and the second side line of the local recognition frame, and using the distance as the width information of the target gap; the first side line is used to identify the position of the table top in the end surface on one side of the expansion joint; the second side line is used to identify the position of the table top in the end surface on the other side of the expansion joint.

[0012] In some embodiments of the first aspect of the present invention, the target gap includes a table top and an inclined surface in both side end surfaces, respectively, and the table top in one side end surface and the inclined surface in the other side end surface are distributed opposite to each other, then the method for extracting the distance between the first side line and the second side line of the local identification frame includes: extracting a first focus between the first side line of the local identification frame and the side line of one side of the target gap, and extracting a second focus between the second side line of the local identification frame and the side line of the other side of the target gap; extracting a first image coordinate of the first focus along the extension direction of the track, and extracting a second image coordinate of the second focus along the extension direction of the track; obtaining the difference between the first coordinate and the second coordinate, and using the difference as the distance between the first side line and the second side line of the local identification frame.

[0013] In some embodiments of the first aspect of the present invention, extracting the depth information of the target gap in the current depth image based on the recognition range of the target gap includes: taking the recognition range of the target gap in the optical image as the first recognition range; obtaining a coordinate mapping relationship between the optical image and the depth image corresponding to the optical image; converting the first recognition range into a second recognition range of the target gap in the depth image according to the coordinate mapping relationship; extracting the depth value corresponding to each pixel point within the second recognition range; and obtaining the depth information corresponding to the target gap using a statistical method based on the depth value corresponding to each pixel point.

[0014] In some embodiments of the first aspect of the present invention, the coordinate mapping relationship is a size ratio between a size of the optical image and a size of the depth image; the identification range is an identification frame; and converting the first identification range into a second identification range of the target gap in the depth image according to the coordinate mapping relationship includes: converting corner point coordinate information of the identification frame in the optical image into corner point coordinate information of the identification frame in the depth image according to the size ratio, which is:

[0015] x_depth = x_src * scale_x;

[0016] y_depth = y_src * scale_y;

[0017] w_depth = w_src * scale_x;

[0018] h_depth = h_src * scale_y;

[0019] scale_x = src_width / detect_width;

[0020] scale_y = src_height / detect_height;

[0021] Among them, scale_x is the coordinate projection coefficient in the first axis direction, scale_y is the coordinate projection coefficient in the second axis direction; src_width is the size of the optical image in the first axis direction; detect_width is the size of the depth image in the first axis direction; src_height is the size of the optical image in the second axis direction; detect_height is the size of the depth image in the second axis direction; {x_src, y_src, w_src, h_src} are the corner point coordinate information of the identification box in the optical image; {x_depth, y_depth, w_depth, h_depth} are the corner point coordinate information of the identification box in the depth image.

[0022] In some embodiments of the first aspect of the present invention, the deep learning-based track gap anomaly detection method, before using the trained machine learning model to perform target gap identification on the current optical image, also includes: performing data screening on the image pairs corresponding to each acquisition moment to eliminate image pairs whose track length is less than a length threshold.

[0023] In some embodiments of the first aspect of the present invention, after performing abnormality detection of the target gap based on the width information and depth information of the target gap, the deep learning-based track gap abnormality detection method further includes: when the target gap is detected as an abnormal gap, performing abnormality level judgment on the abnormal gap, so as to perform corresponding processing according to the abnormality level judgment result.

[0024] The present invention provides a track gap anomaly detection device based on deep learning in a second aspect, comprising: an optical image acquisition unit, used to acquire each optical image of the track during the track inspection process; a depth image acquisition unit, used to acquire each depth image of the track during the track inspection process; an anomaly detection unit, connected to the optical image acquisition unit and the depth image acquisition unit, to repeatedly perform the target gap anomaly detection process based on the image pair acquired at each acquisition moment; wherein the image pair includes a depth image and an optical image; and the anomaly detection unit comprises: an identification subunit, used to use a trained machine learning model to perform target gap recognition on the current optical image, and when the target gap is included in the current optical image, extract the recognition range of the target gap in the current optical image; a width extraction subunit, used to extract the width information of the target gap in the current optical image according to the recognition range of the target gap; a depth extraction subunit, used to extract the depth information of the target gap in the current depth image according to the recognition range of the target gap; a detection subunit, used to perform target gap anomaly detection according to the width information and depth information of the target gap, so as to obtain the anomaly detection result corresponding to each acquisition moment.

[0025] The present invention provides a terminal in a third aspect, comprising: a processor and a memory, wherein the memory is communicatively connected to the processor; the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the terminal executes any of the deep learning-based track gap anomaly detection methods described above.

[0026] In a fourth aspect, the present invention provides a computer storage medium storing a computer program, which, when executed by a processor, implements any of the deep learning-based track gap anomaly detection methods described above.

[0027] As described above, the track gap anomaly detection method, device, terminal and computer storage medium based on deep learning, through real-time acquisition of depth images and optical images at each acquisition moment, to perform target gap recognition based on optical images, and through extraction of width information and depth information of the target gap, to perform target gap anomaly determination based on the width information and depth information, thereby being able to quickly and accurately obtain the target gap anomaly detection results corresponding to each acquisition moment. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 Shown is a schematic structural diagram of the inspection vehicle of the present invention in one embodiment;

[0029] Figure 2 Shown is a schematic flow chart of a track gap anomaly detection method based on deep learning in one embodiment of the present invention;

[0030] Figure 3 It is a schematic flow chart of an abnormality detection process of a target gap in one embodiment of the present invention;

[0031] Figure 4 It is a schematic flow chart showing the abnormality detection process of the target gap according to another embodiment of the present invention;

[0032] Figure 5 Shown is a schematic diagram of a flow chart of extracting width information of a target gap in a current optical image according to the identification range of the target gap in an embodiment of the present invention;

[0033] Figure 6 Shown is a schematic diagram of a flow chart of extracting width information of a target gap in a current optical image according to the identification range of the target gap in another embodiment of the present invention;

[0034] Figure 7 Shown is a schematic diagram of the expansion joint described in the present invention in one embodiment; Figure 8Shown are schematic diagrams of the expansion joint described in the present invention in other embodiments;

[0035] Fig. 9 Shown is a schematic diagram of the local identification frame in one embodiment of the present invention;

[0036] Fig.10 It is a schematic diagram of the process of step S200 in one embodiment of the present invention;

[0037] Fig.11 Shown is a schematic flow chart of extracting depth information of a target gap in a current depth image according to the recognition range of the target gap in another embodiment;

[0038] Fig.12 Shown is a schematic flow chart of a track gap anomaly detection method based on deep learning in another embodiment of the present invention;

[0039] Fig.13 Shown is a schematic diagram of the structure of a track gap anomaly detection device based on deep learning in one embodiment of the present invention;

[0040] Fig.14 Shown is a schematic diagram of the structure of the abnormality detection unit in one embodiment of the present invention;

[0041] Fig.15 Shown is a schematic structural diagram of the electronic terminal in one embodiment of the present invention. DETAILED DESCRIPTION

[0042] The following describes the embodiments of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0043] It should be noted that in the following description, reference is made to the accompanying drawings, which describe several embodiments of the present application. It should be understood that other embodiments may be used, and mechanical, structural, electrical, and operational changes may be made without departing from the spirit and scope of the present application.

[0044] Also, as used herein, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context indicates otherwise. It should be further understood that the terms "comprises", "includes", "includes" indicate the presence of the described features, operations, elements, components, items, kinds, and / or groups, but do not exclude the presence, occurrence, or addition of one or more other features, operations, elements, components, items, kinds, and / or groups. The terms "or" and "and / or" used herein are interpreted as inclusive, or mean any one or any combination.

[0045] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the technical solution in the embodiments of the present invention is further described in detail through the following embodiments and in combination with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the invention.

[0046] In order to solve the technical problems existing in the prior art, the present application first provides a track gap anomaly detection method based on deep learning, which is used to perform abnormal detection of target gaps on the track during track inspection to obtain the abnormal detection results of the target gaps corresponding to each moment.

[0047] The target gap is a gap caused by loss / wear of the track, including but not limited to one or more of scratches, misaligned gaps, worn gaps or other damaged gaps.

[0048] In a specific embodiment, the track gap anomaly detection method based on deep learning is applicable to a track inspection vehicle; the inspection vehicle is as follows: Figure 1 As shown, it includes: an inspection vehicle body 1, a detection camera 2, a control host 3, a driving component (not shown) and a processor (not shown).

[0049] The inspection vehicle body 1 is connected to a driving member so as to run along a track under the driving action of the driving member.

[0050] Among them, the detection camera 2 includes a depth acquisition module and an optical acquisition module; the control host 3 is connected to the detection camera to control the depth acquisition module and the optical acquisition module to respectively perform the acquisition process to obtain the depth image and optical image corresponding to each acquisition moment.

[0051] The depth acquisition module and the optical acquisition module are respectively connected to the processor to transmit the acquired depth image and optical image to the processor.

[0052] The processor processes the received depth image and the optical image to obtain an abnormality determination result corresponding to each acquisition moment.

[0053] See also Figure 2, showing a flow chart of the deep learning-based track gap anomaly detection method provided in the present application in one embodiment.

[0054] like Figure 2 As shown, the method comprises the following steps:

[0055] S100, during the inspection process, performing image acquisition of track depth images and optical images to obtain image pairs consisting of depth images and optical images at each acquisition moment;

[0056] Wherein, the depth image is an image containing depth information corresponding to each pixel point on the track surface;

[0057] The optical image is an image containing the optical radiation intensity corresponding to each pixel point on the track surface.

[0058] In an optional embodiment, the depth image is a three-dimensional point cloud image, and the optical image is an optical intensity image.

[0059] Specifically, during the inspection process, a first image acquisition process of the track depth image is performed using a depth acquisition module, and a second image acquisition process of the track optical image is performed using an optical acquisition module to obtain the depth image and optical image corresponding to each acquisition moment; the depth image and optical image corresponding to the same acquisition moment are constructed into an image pair; based on this method, the image pair corresponding to each acquisition moment is obtained.

[0060] It should be noted that the first image acquisition process and the second image acquisition process may be performed synchronously or asynchronously.

[0061] In some embodiments, when the first image acquisition process and the second image acquisition process are executed asynchronously, for each acquired depth image, an optical image at a time closest to the acquisition time of the current depth image is extracted, and an image pair is constructed based on the optical image and the current depth image; or similarly, for each acquired optical image, a depth image at a time closest to the acquisition time of the current optical image is extracted, and an image pair is constructed based on the depth image and the current optical image.

[0062] In order to obtain image pairs more quickly and conveniently, in some embodiments, during the driving process of the inspection vehicle, a control host is used to control the depth acquisition module and the optical acquisition module on the inspection vehicle to capture the depth image and the optical image of the track surface at the same time; the depth image and the optical image corresponding to each acquisition moment are taken as the image pair corresponding to the acquisition moment, and uploaded to the processor for processing through the WiFi module.

[0063] S200, during the image acquisition process, based on the image pairs at each acquisition moment, respectively performing an abnormality detection process of the target gap to obtain an abnormality detection result corresponding to each acquisition moment.

[0064] Specifically, for a single acquisition moment, the abnormality detection process of the target gap is as follows: Figure 3 As shown, including:

[0065] S201, using the trained machine learning model to perform target gap recognition on the current optical image;

[0066] The current target optical image is the optical image corresponding to the current acquisition moment.

[0067] The machine learning model is a model obtained by training a preset convolution model using several groups of sample data pairs corresponding to the target gap, and is used to identify the target gap; the sample data pairs include an optical image corresponding to the target gap and a pre-marked identification box range of the same target gap.

[0068] Specifically, the trained machine learning model is used to perform target gap recognition on the current optical image to obtain a recognition result corresponding to the current optical image, that is, to determine whether the current optical image contains the target gap.

[0069] In order to improve the recognition accuracy and efficiency of the model, in an optional embodiment, the loss number of the machine learning model is a function constructed based on recall and accuracy; wherein the recall is the total number of abnormal targets detected in the test samples divided by the total number of abnormalities in all test samples; and the accuracy is the ratio between the number of sample images in which the target gap is accurately identified and the total number of sample images.

[0070] More specifically, the Loss function used by the machine learning model is:

[0071] Loss = a L classifer + (1- a ) L iou ;

[0072] in, a is the weight coefficient, which is a numerical value between (0,1) and 0.5.

[0073]

[0074] in, N is the number of samples in the training set; L classifer is the classification category loss function of the target gap sample, which is used to characterize the recognition recall rate of the target gap sample; Liou is the IOU loss function, which is used to characterize the recognition range loss function of the target gap sample; L i is the loss function of the i-th target gap sample; y i For the i The identification value of the target gap sample; p i For the i The probability value of the target gap sample; is the learning rate; v is the learning step length; b and b gt Respectively represent the position of the recognition range and the position of the corresponding identification range of the target gap sample; c It is the distance between the recognition range and the identification range corresponding to the target gap sample.

[0075] S202, determining whether the current optical image contains a target gap;

[0076] Specifically, after the target gap is identified on the current optical image, it is determined whether the current optical image includes the target gap.

[0077] S203A, if no, then exit the anomaly detection process at the current collection moment, and execute the anomaly detection process at the next collection moment;

[0078] S203B, if yes, extracting the recognition range of the target gap;

[0079] The identification range is the identification frame range corresponding to the identified target gap.

[0080] Specifically, when the trained machine learning model is used to identify the target gap in the current optical image, the range coordinates of the identification frame corresponding to the target gap are extracted as the identification range of the target gap.

[0081] S204, extracting width information of the target gap from the current optical image and extracting depth information of the target gap from the current depth image according to the recognition range of the target gap;

[0082] Wherein, the current depth image is a depth image acquired at the current acquisition moment, which corresponds to the current optical image;

[0083] The width information is used to characterize the distance feature between the target gap and the two side lines, including but not limited to one or more of the distance mean, maximum value, minimum value and total value;

[0084] Likewise, the depth information is a depth feature used to characterize the target gap, including but not limited to one or more of a mean value, a maximum value, a minimum value, and a total value of the depth of the target gap.

[0085] Specifically, after determining the recognition range of the target gap, the width information of the target gap within the recognition range is extracted; and based on the recognition range, the distribution area of ​​the target gap in the target depth image and the position information of the distribution area are determined, so as to extract the depth information of the target gap within the distribution area based on the position information of the distribution area as the depth information of the target gap.

[0086] S205, performing abnormality detection of the target gap according to the width information and depth information of the target gap;

[0087] Specifically, a width threshold and a depth threshold of the target gap are obtained;

[0088] Based on the width threshold and the width information, performing a width abnormality determination of the target gap; illustratively, when the width information of the target gap is greater than the width threshold, determining that the target gap has a width abnormality;

[0089] And, based on the depth threshold and the depth information, performing depth anomaly determination of the target gap; illustratively, when the depth information of the target gap is greater than the depth threshold, determining that the target gap is depth anomaly;

[0090] When it is detected that the target gap is either abnormal in width or abnormal in depth, the target gap is determined to be an abnormal gap, and a corresponding abnormal warning is executed.

[0091] In some optional embodiments, the target gap abnormality detection process, after executing step S205, is as follows: Figure 4 As shown, it also includes:

[0092] S206, when the target gap is detected as an abnormal gap, an abnormality level determination is performed on the abnormal gap, and corresponding processing is performed according to the abnormality level determination result.

[0093] Specifically, after an abnormal gap is detected, an abnormal level determination of the abnormal gap is performed according to a preset abnormal level threshold range to obtain a corresponding abnormal level and a corresponding gap processing solution, and the abnormal level and the corresponding gap processing solution are pushed to the user end;

[0094] Optionally, after obtaining the abnormality level corresponding to the abnormal gap, the identification range and abnormality level of the abnormal gap may be displayed on the user's display to facilitate the user to visually judge the target gap.

[0095] When the abnormal detection process of the target gap is performed by the inspection vehicle in an offline state, that is, when the machine learning model is a processing unit mounted in the inspection vehicle, in order to improve the recognition efficiency and accuracy of the target gap, and take into account the memory occupancy rate of the inspection vehicle processing unit due to the model size;

[0096] In some optional embodiments, the machine learning model collects a Yolo model, performs pre-screening on the optical image of the track to obtain an optical image containing a target gap as a sample image, and performs annotation on the sample image to obtain an identification box corresponding to the target gap in each sample image as an identification box corresponding to the sample image; based on the sample image and the identification box corresponding to the sample image, a sample pair for training the model is constructed; using each sample pair, model training and hyperparameter tuning are performed on the pre-constructed initial Yolo model to obtain a trained Yolo model, and the trained Yolo model is used as a machine learning model for identifying target gaps.

[0097] Based on this, the Yolo model is used to identify the target gap, including:

[0098] Using a first convolutional neural network to extract features from an input optical image; wherein the deep convolutional neural network comprises a plurality of convolutional layers and pooling layers for extracting feature information from the optical image;

[0099] Generate the candidate region of interest (ROI) of the target gap using the Region Proposal Network (RPN) method;

[0100] For each of the candidate regions, performing feature extraction using a second convolutional network (object detection branch) to determine the location and size of the target gap based on the extracted features;

[0101] After determining the position and size of the target gap, the extracted target gap is classified and the recognition frame is adjusted; wherein the classification is to compare the prediction result of the target gap with the category threshold to determine whether it is a target gap; the recognition frame adjustment is to adjust the position and size of the recognition frame according to the position and size information of the target gap to make it more suitable for the application scenario.

[0102] In a more specific example, the machine learning model collects the Yolov8 model so that the model can run more smoothly on a mobile device. At the same time, while maintaining high performance, the model also has high real-time performance and strong generalization ability, and can adapt to target gaps of different sizes and shapes.

[0103] To ensure the accuracy of the target gap width information and reduce the influence of the non-target gap area on the extracted width information; in some optional embodiments, the target gap width information is the width information of the target gap within the recognition range, then the implementation method of extracting the target gap width information in the current optical image according to the recognition range of the target gap is as follows: Figure 5 As shown, including;

[0104] S801, based on the recognition range of the target gap, extracting a target section of the target gap within the recognition range from the optical image;

[0105] S802 , extracting vertical distance values ​​of each pixel point on both side lines of the target gap in the target section, so as to obtain width information of the target gap based on each distance value.

[0106] In a specific embodiment, when the width information is the minimum width of the target gap between the two side lines, after extracting the distance value between each pixel point and the other side line, the minimum value of the distance value corresponding to each pixel point is taken, and the minimum value is used as the minimum width of the target gap.

[0107] In a specific embodiment, when the width information is the average width of the target gap between the two side lines, after extracting the distance value between each pixel point and the other side line, the distance value corresponding to each pixel point is averaged, and the average is used as the average width of the target gap.

[0108] More specifically, the method for obtaining the vertical distance value of each pixel point includes:

[0109] One side line of the target gap is used as the current side line; the vertical direction of each pixel point of the current side line is extracted; and along the vertical direction, the distance between each pixel point in the current side line and the other side line is extracted.

[0110] Since there are many irregular gaps in the track, that is, the boundary lines on both sides of the gap are not parallel, in order to further improve the extraction efficiency of the width information, the implementation method of extracting the width information of the target gap in the current optical image according to the recognition range of the target gap is as follows: Figure 6 As shown, including:

[0111] S801′, based on the recognition range of the target gap, extracting a target section of the target gap within the recognition range from the optical image;

[0112] S802', performing linear fitting on the first side edge line and the second side edge line of the target gap in the target section respectively to obtain corresponding fitting lines;

[0113] S803', determining the vertical direction of each fitting line, extracting the distance value between each pixel point on the fitting line and the other side line along the vertical direction, so as to obtain the width information of the target gap based on each distance value.

[0114] When the target gap is an expansion joint, according to the process requirements of the rail expansion joint, a table is often formed in the rail end faces on both sides of the expansion joint; for example, Figure 7 As shown, the expansion joint includes a first table 811 and an inclined surface 812 in the lower rail end surface (hereinafter referred to as the "lower end surface") 810 in the figure, and the expansion joint includes a first table 821 and an inclined surface 822 in the upper rail end surface (hereinafter referred to as the "upper end surface") 820 in the figure; wherein the first table is the area located at the endmost part of the lower end surface or the upper end surface; the first table 811 in the lower end surface and the inclined surface 822 in the upper end surface are distributed opposite to each other, and the first table 821 in the upper end surface and the inclined surface 812 in the lower end surface are distributed opposite to each other.

[0115] It should be noted that, in some other embodiments, a plurality of stepped table surfaces are formed in the rail end surfaces on both sides of the expansion joint, and the table surfaces on both sides are oppositely distributed and the sizes and shapes of the two are matched; for example, the rail end surfaces on both sides of the expansion joint are as follows: Figure 8 The formation shown has two terraces, namely Figure 8 The lower end surface of the middle expansion joint includes a first table surface 811, an inclined surface 812 and a second table surface 813; the upper end surface of the expansion joint 800 includes a first end surface 821, an inclined surface 822 and a third table surface 823; the first table surface 811 in the lower end surface is arranged opposite to the second end surface 823 in the upper end surface, and the second table surface 813 in the lower end surface is arranged opposite to the first table surface 821 in the upper end surface.

[0116] For the above-mentioned expansion joint, since the gap width between the expansion joint and the table top and the gap width between the expansion joint and the inclined surface are often different; in order to quickly and accurately obtain the width information of the expansion joint, the identification frame identified and extracted by the machine learning model is used, and a local identification frame covering the local area of ​​the expansion joint is adopted; wherein, the local identification frame is used to identify the enclosed area enclosed by the table tops in the end surfaces on both sides of the expansion joint, that is, to identify the enclosed area enclosed by the first table top and the second table top.

[0117] by Figure 7 Taking the expansion joint shown as an example, in the optical image, Fig. 9 As shown, the first side line (upper side line) 910 of the local identification frame is used to identify the position of the first table top in the upper end surface; the second side line (lower side line) 920 of the local identification frame is used to identify the position of the first table top in the first side end surface (lower side end surface) of the expansion joint; the first side line and the second side line are parallel to each other.

[0118] Based on this, when step S203 is executed, the implementation method of extracting the width information of the target gap in the current optical image according to the recognition range of the target gap includes:

[0119] Extracting a first edge line in the local identification frame, and extracting a second edge line in the local identification frame;

[0120] The distance between the first sideline and the second sideline of the local recognition frame is extracted, and the distance is used as the width information of the target gap.

[0121] More specifically, in the optical image, Fig. 9 As shown, the first side line of the local recognition frame and the second side line of the target gap (that is, the second side end surface at Fig. 9 The first focus between the ridge line in the view) and the second edge line of the local recognition frame and the first side edge line of the target gap (that is, the first side end surface at Fig. 9 The second focus between the ridge lines under the view;

[0122] Extracting first image coordinates of the first focus along the extending direction of the track, and extracting second image coordinates of the second focus along the extending direction of the track;

[0123] The difference between the first coordinate and the second coordinate is obtained, and the difference is used as the distance between the upper and lower edges of the local recognition frame.

[0124] It should be noted that in order to achieve accurate recognition of the local identification frame, the sample data pair used by the machine learning model used in step S201 during model training includes an optical image corresponding to the expansion joint and a local identification frame range corresponding to the expansion joint; the local identification frame range is the range of the enclosed area enclosed by the table tops on both side end faces of the expansion joint.

[0125] In some optional embodiments, before the step S204 is performed, Fig.10 As shown, the step S200 further includes:

[0126] S207, performing coordinate alignment on the current target image and the current depth image so that the two have the same coordinate system;

[0127] After the recognition range of the target gap is obtained, the depth information of the target gap within the recognition range in the current depth image is directly extracted based on the recognition range.

[0128] However, when the coordinate alignment is not achieved between the current target image and the current depth image, or when the size of the current target image is different from the size of the current depth image, the implementation method of extracting the depth information of the target gap in the current depth image according to the recognition range of the target gap is as follows: Fig.11 As shown, including:

[0129] S901, obtaining a recognition range of a target gap in an optical image as a first recognition range;

[0130] S902, acquiring a coordinate mapping relationship between an optical image and a depth image corresponding to the optical image;

[0131] S903, converting the first recognition range into a second recognition range of the target gap in the depth image according to the coordinate mapping relationship;

[0132] S904, extracting the depth value corresponding to each pixel point within the second recognition range; and obtaining the depth information corresponding to the target gap using a statistical method based on the depth value corresponding to each pixel point.

[0133] More specifically, after determining the identification frame corresponding to the target gap, the corner point coordinate information {x_src, y_src, w_src, h_src} of the identification frame in the optical image is obtained as the first identification range of the target gap; wherein, x_src and y_src are respectively the first axial coordinate and the second axial coordinate of the lower left corner of the identification frame, and w_src and h_src are the first axial coordinate and the second axial coordinate of the upper right corner of the identification frame; the first axial direction and the second axial direction are the first direction and the second direction perpendicular to each other in the coordinate system where the target gap is located; exemplarily, the first axial direction is the direction perpendicular to the extension direction of the track, and the second axial direction is the direction parallel to the extension of the track.

[0134] According to the size of the optical image and the size of the depth image, the coordinate projection coefficient between the optical image and the depth image is determined as the coordinate mapping relationship:

[0135] scale_x = src_width / detect_width;

[0136] scale_y = src_height / detect_height.

[0137] Among them, scale_x is the coordinate projection coefficient in the first axis direction, scale_y is the coordinate projection coefficient in the second axis direction; src_width is the size of the optical image in the first axis direction; detect_width is the size of the depth image in the first axis direction; src_height is the size of the optical image in the second axis direction; detect_height is the size of the depth image in the second axis direction.

[0138] According to the coordinate projection coefficient, the corner point coordinate information of the identification frame in the optical image is converted into the corner point coordinate information {x_depth, y_depth, w_depth, h_depth} of the identification frame in the depth image as the second identification range of the target gap; wherein,

[0139] x_depth = x_src * scale_x;

[0140] y_depth = y_src * scale_y;

[0141] w_depth = w_src * scale_x;

[0142] h_depth = h_src * scale_y;

[0143] According to the second identification range of the target gap in the depth image, the depth value corresponding to each pixel point within the second identification range is extracted respectively; based on the depth value corresponding to each pixel point, the depth information corresponding to the target gap is obtained using a statistical method.

[0144] In a specific embodiment, when the target gap is a malocclusion gap, the depth information is the vertical malocclusion depth, that is, the depth difference between the maximum depth value and the minimum depth value within the second recognition range, which is:

[0145] S(x,y,w,h) =max(x,y,w,h)—min(x,y,w,h);

[0146] Among them, 0≤x≤x_depth, 0≤y≤y_depth, 0≤w≤w_depth, 0≤h≤ h_depth.

[0147] When the depth information is the maximum depth, it is:

[0148] In a specific embodiment, when the target gap is a wear gap, the depth information is the maximum depth value within the second identification range, which is:

[0149] S(x,y,w,h) =max(x,y,w,h)

[0150] Among them, 0≤x≤x_depth, 0≤y≤y_depth, 0≤w≤w_depth, 0≤h≤ h_depth.

[0151] To improve the recognition efficiency of the target gap, in some optional embodiments, the track gap anomaly detection method based on deep learning is performed before executing step S200, such as Fig.12 As shown, it also includes:

[0152] S300, performing data screening on the image pairs corresponding to each acquisition moment to eliminate image pairs whose track length is less than a length threshold.

[0153] Specifically, for the image pair corresponding to each acquisition moment, the first length of the track contained in the optical image in the image pair and the second length of the track contained in the depth image in the image pair are extracted respectively; it is detected whether the first length is less than a length threshold and / or whether the first length is less than a length threshold. If so, it is determined that the length of the track contained in the image pair is less than the length threshold, and the image pair is discarded.

[0154] In order to solve the technical problems existing in the prior art, the present invention also provides a track gap anomaly detection device based on deep learning in the second aspect, which is used to perform abnormal detection of target gaps on the track during track inspection to obtain the abnormal detection results of the target gaps corresponding to each moment.

[0155] See also Fig.13 , which is a schematic diagram of the structure of the track gap anomaly detection device based on deep learning in one embodiment, as shown in Fig.13 As shown, the track gap anomaly detection device 10 includes: an optical image acquisition unit 11, a depth image acquisition unit 12 and an anomaly detection unit 13.

[0156] Wherein, the optical image acquisition unit 11 is used to acquire various optical images of the track during the track inspection process;

[0157] The depth image acquisition unit 12 is used to acquire various depth images of the track during the track inspection process;

[0158] The abnormality detection unit 13 is connected to the optical image acquisition unit and the depth image acquisition unit to obtain image pairs corresponding to each acquisition moment, so as to repeatedly perform the abnormality detection process of the target gap based on the image pairs acquired at each acquisition moment. The image pairs include depth images and optical images.

[0159] In a specific embodiment, the abnormality detection unit is as follows: Fig.14 As shown, it includes: a recognition subunit 131, a width extraction subunit 132, a depth extraction subunit 133 and a detection subunit 134;

[0160] The recognition subunit 131 is used to use the trained machine learning model to perform target gap recognition on the current optical image, and when the current optical image contains the target gap, extract the recognition range of the target gap in the current optical image;

[0161] The width extraction subunit 132 is used to extract the width information of the target gap in the current optical image according to the recognition range of the target gap;

[0162] The depth extraction subunit 133 extracts the depth information of the target gap in the current depth image according to the recognition range of the target gap;

[0163] And the detection subunit 134 is used to perform abnormality detection of the target gap according to the width information and depth information of the target gap to obtain the abnormality detection result corresponding to each acquisition time.

[0164] It should be noted that when the above units and sub-units perform corresponding functions, their corresponding execution steps are the same as the implementation methods in the above embodiments, and will not be repeated here.

[0165] Based on the same technical concept, the track gap anomaly detection method based on deep learning provided in the above embodiment of the present invention can be implemented on the terminal side or the server side.

[0166] See also Fig.15 , is an optional hardware structure diagram of an electronic terminal 700 provided in an embodiment of the present invention. The electronic terminal 700 may be a live broadcast machine, a camera, a mobile phone, a computer device, a tablet device, a personal digital processing device, a factory background processing device, etc. that integrates photo / video functions. The electronic terminal 700 includes: at least one processor 701, a memory 702, at least one network interface 704 and a user interface 706. The various components in the device are coupled together through a bus system 705. It can be understood that the bus system 705 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 705 also includes a power bus, a control bus and a status signal bus.

[0167] The user interface 706 may include a display, a keyboard, a mouse, a trackball, a click gun, keys, buttons, a touch pad or a touch screen.

[0168] It is understood that the memory 702 can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), which is used as an external cache. By way of exemplary but not limiting explanation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM). The memory described in the embodiments of the present invention is intended to include but is not limited to these and any other suitable categories of memory.

[0169] The memory 702 in the embodiment of the present invention is used to store various categories of data to support the operation of the electronic terminal 700. Examples of these data include: any executable program used to operate on the electronic terminal 700, such as an operating system 7021 and an application 7022; the operating system 7021 includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks. The application 7022 may include various applications, such as a media player (MediaPlayer), a browser (Browser), etc., for implementing various application services. The deep learning-based track gap anomaly detection method implementing the embodiment of the present invention may be included in the application 7022.

[0170] The method disclosed in the above embodiment of the present invention can be applied to the processor 701, or implemented by the processor 701. The processor 701 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 701 or the instruction in the form of software. The above processor 701 may be a general processor, a digital signal processor (DSP, Digital Signal Processor), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor 701 can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiment of the present invention. The general processor 701 may be a microprocessor or any conventional processor, etc. In combination with the steps of the accessory optimization method provided in the embodiment of the present invention, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium, which is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0171] In an exemplary embodiment, the electronic terminal 700 may be implemented by one or more application specific integrated circuits (ASIC), DSP, programmable logic device (PLD), complex programmable logic device (CPLD) to execute the aforementioned method.

[0172] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when called by a processor, implements the track gap anomaly detection method based on deep learning provided by the present invention.

[0173] Among them, the computer-readable storage medium can be a tangible device that can hold and store instructions used by the instruction execution device. The computer-readable storage medium can be, for example, (but not limited to) an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, and a mechanical encoding device.

[0174] The computer-readable program described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. A network adapter card or a network interface in each computing / processing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in each computing / processing device.

[0175] It should be noted that, in the various embodiments of the present application, the serial numbers of the above steps do not indicate the order of execution. The execution order of the steps should be determined by their functions and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0176] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. A track gap anomaly detection method based on deep learning, characterized in that: Applicable to a processor; during a track inspection process, based on the image pairs collected at each collection time, repeatedly performing an abnormality detection process of a target gap; wherein the image pairs include a depth image and an optical image; For a single acquisition moment, the track gap anomaly detection method based on deep learning includes: Using the trained machine learning model, the target gap is identified on the current optical image; when the current optical image is determined to contain the target gap, the following steps are performed: Extracting the recognition range of the target gap in the current optical image; Extracting width information of the target gap in the current optical image according to the recognition range of the target gap, and extracting depth information of the target gap in the current depth image according to the recognition range of the target gap; Based on the width information and depth information of the target gap, perform anomaly detection of the target gap to obtain anomaly detection results corresponding to each acquisition time; When the target gap is an expansion gap, the method for extracting the width information of the target gap in the current optical image according to the recognition range of the target gap includes: The distance between the first sideline and the second sideline of the local identification frame is extracted, and the distance is used as the width information of the target gap; wherein the local identification frame is used to identify the enclosed area enclosed by the table tops in the end faces on both sides of the expansion joint; the first sideline is used to identify the position of the table top in the end face on one side of the expansion joint; and the second sideline is used to identify the position of the table top in the end face on the other side of the expansion joint.

2. The track gap anomaly detection method based on deep learning according to claim 1, characterized in that: The loss function of the machine learning model is a function constructed based on the classification category loss function and the recognition range loss function of the target gap sample, which is: Loss=aL classifer +(1-a)L iou ; Among them, a is the weight coefficient; N is the number of training set samples; L classifer is the classification category loss function of the target gap sample; L iou is the recognition range loss function of the target gap sample; L i is the loss function of the i-th target gap sample; y i is the identification value of the i-th target gap sample; p i is the probability value of the i-th target gap sample; ρ is the learning rate; v is the learning step size; b and b gt They respectively represent the position of the recognition range and the position of the corresponding identification range corresponding to the target gap sample; c is the distance between the recognition range and the identification range corresponding to the target gap sample.

3. The track gap anomaly detection method based on deep learning according to claim 1, characterized in that: The method for extracting the width information of the target gap in the current optical image according to the recognition range of the target gap includes: Based on the recognition range, extracting a target section of a target gap within the recognition range in the optical image; In the target section, extracting the vertical distance value of each pixel point on the two side lines of the target gap, so as to obtain the width information of the target gap based on each distance value; The width information is used to characterize the distance feature between the target gap and the side lines on both sides.

4. The track gap anomaly detection method based on deep learning according to claim 3 is characterized in that: When the target gap is an irregular gap, the method for extracting the vertical distance value of each pixel point on the two side lines of the target gap in the target section includes: In the target section, linear fitting is performed on the first side edge line and the second side edge line of the target gap respectively to obtain corresponding fitting lines; The vertical direction of each fitting line is determined, and the distance value between each pixel point on the fitting line and the other side line is extracted along the vertical direction.

5. The track gap anomaly detection method based on deep learning according to claim 1, characterized in that: The target gap includes a table surface and an inclined surface in the two side end surfaces respectively, and the table surface in one side end surface and the inclined surface in the other side end surface are distributed opposite to each other, then the method for extracting the distance between the first side line and the second side line of the local identification frame includes: Extracting a first focus between a first sideline of the local identification frame and a sideline of the target gap, and extracting a second focus between a second sideline of the local identification frame and another sideline of the target gap; Extracting first image coordinates of the first focus along the extending direction of the track, and extracting second image coordinates of the second focus along the extending direction of the track; The difference between the first coordinate and the second coordinate is obtained, and the difference is used as the distance between the first sideline and the second sideline of the local recognition frame.

6. The track gap anomaly detection method based on deep learning according to claim 1, characterized in that: The step of extracting depth information of the target gap from the current depth image according to the recognition range of the target gap includes: Taking the recognition range of the target gap in the optical image as a first recognition range; Acquire a coordinate mapping relationship between an optical image and a depth image corresponding to the optical image; According to the coordinate mapping relationship, converting the first recognition range into a second recognition range of the target gap in the depth image; Extracting the depth value corresponding to each pixel point within the second recognition range; and obtaining the depth information corresponding to the target gap using a statistical method based on the depth value corresponding to each pixel point.

7. The track gap anomaly detection method based on deep learning according to claim 6, characterized in that: The coordinate mapping relationship is the size ratio between the size of the optical image and the size of the depth image; the recognition range is the recognition frame; The converting the first recognition range into a second recognition range of the target gap in the depth image according to the coordinate mapping relationship includes: According to the size ratio, the corner point coordinate information of the identification frame in the optical image is converted into the corner point coordinate information of the identification frame in the depth image, which is: x_depth = x_src * scale_x; y_depth=y_src*scale_y; w_depth = w_src * scale_x; h_depth=h_src*scale_y; scale_x=src_width / detect_width; scale_y=src_height / detect_height Among them, scale_x is the coordinate projection coefficient in the first axis direction, scale_y is the coordinate projection coefficient in the second axis direction; src_width is the size of the optical image in the first axis direction; detect_width is the size of the depth image in the first axis direction; src_height is the size of the optical image in the second axis direction; detect_height is the size of the depth image in the second axis direction; {x_src, y_src, w_src, h_src} are the corner point coordinate information of the identification box in the optical image; {x_depth, y_depth, w_depth, h_depth} are the corner point coordinate information of the identification box in the depth image.

8. The track gap anomaly detection method based on deep learning according to claim 1, characterized in that: After performing abnormality detection of the target gap based on the width information and the depth information of the target gap, the method further includes: When the target gap is detected as an abnormal gap, an abnormality level determination is performed on the abnormal gap, so as to perform corresponding processing according to the abnormality level determination result.

9. A track gap anomaly detection device based on deep learning, characterized in that: include: An optical image acquisition unit, used to acquire optical images of the track during the track inspection process; A depth image acquisition unit, used to acquire various depth images of the track during the track inspection process; an abnormality detection unit, connected to the optical image acquisition unit and the depth image acquisition unit, to repeatedly perform an abnormality detection process of a target gap based on an image pair acquired at each acquisition moment; wherein the image pair includes a depth image and an optical image; and The abnormality detection unit comprises: an identification subunit, configured to use the trained machine learning model to perform target gap identification on the current optical image, and to extract an identification range of the target gap in the current optical image when the current optical image contains the target gap; A width extraction subunit, configured to extract width information of the target gap in the current optical image according to the recognition range of the target gap; A depth extraction subunit, which extracts depth information of the target gap in the current depth image according to the recognition range of the target gap; A detection subunit, configured to perform abnormality detection of the target gap according to the width information and the depth information of the target gap, so as to obtain an abnormality detection result corresponding to each acquisition time; When the target gap is an expansion gap, the method for extracting the width information of the target gap in the current optical image according to the recognition range of the target gap includes: The distance between the first sideline and the second sideline of the local identification frame is extracted, and the distance is used as the width information of the target gap; wherein the local identification frame is used to identify the enclosed area enclosed by the table tops in the end faces on both sides of the expansion joint; the first sideline is used to identify the position of the table top in the end face on one side of the expansion joint; and the second sideline is used to identify the position of the table top in the end face on the other side of the expansion joint.

10. A terminal, characterized in that: include: A processor and a memory, wherein the memory is communicatively connected to the processor; The memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the terminal executes the track gap anomaly detection method based on deep learning as described in any one of claims 1 to 8.

11. A computer storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the track gap anomaly detection method based on deep learning as described in any one of claims 1 to 8 is implemented.

Images