Remote rail identification and obstacle detection method and system

Abstract

The invention discloses a long-distance rail identification and obstacle detection method and system, and the method comprises the steps: combining a long-focus optical system with an ultra-high-speed industrial camera at a hardware level, and building a high-collection-frame-rate and long-distance optical detection system; on the software level, a region of interest for obstacle detection is delimited according to recognition and visual segmentation of left and right rail lines of a rail, advanced visual tasks such as a rail obstacle intrusion detection algorithm based on target detection are executed, and finally high-frame-rate recognition and detection of long-distance rails and rail obstacles are achieved. According to the invention, a Schmidt-Cassegrain optical system with a focal length of 2000mm is used, and an ultra-high-speed industrial camera with a resolution of 1920 * 1080 and 3000 frames per second is combined to build a long-distance high-frame-rate optical detection platform which is erected on a driving position of a train to detect railway conditions within a range of 6km in real time; serious accidents caused by overlong braking distance due to high-speed running of the train can be prevented. Compared with a traditional train driver eye observation and laser radar-based scanning imaging mode, a farther detection distance, better detection and recognition robustness and a higher frame rate can be realized through the long-distance optical observation platform and the rear-end real-time detection system.

Description

Technical Field

[0001] The present invention belongs to the fields of optical design, computer vision and computer control, and specifically relates to telephoto optical system design, target detection and recognition technology, and real-time video acquisition and processing technology. The present invention is a real-time detection method for long-distance track obstacle intrusion. Background Art

[0002] The patent application with publication number CN116030430A discloses a rail identification method, device, equipment and storage medium. The method fuses the rail data collected by the image recognition device and the radio detection device to obtain rail fusion image data; processes the rail fusion image data to obtain the rail profile image; and extracts the coordinate points of the rail profile image using a sliding window to obtain the target rail position. By fusing the rail data collected by the two sensors, the defects such as low accuracy, easy failure and poor stability caused by the use of a single sensor are avoided, and the all-weather identification and extraction of the rail is realized. However, the rail identification method based on multi-sensor information fusion has high requirements for the calibration accuracy of different sensors. There are certain calibration errors in the actual scene, resulting in poor data fusion effect. At the same time, the sliding window method for obtaining the track coordinate points of the rail profile image has a pixel traversal process, which is more suitable for scenes with low real-time requirements.

[0003] The patent application with publication number CN117292351A discloses a subway obstacle detection system and method based on background subtraction method. The method first pre-collects an obstacle-free image as a background frame and a real-time collected image as a frame to be detected; extracts the feature vectors of both through a convolutional neural network and a pooling method and performs vectorized encoding, generates a retrieval codebook based on the vectorized encoding of the background frame, and obtains the corresponding frame by referring to the retrieval codebook after the vectorized encoding of the frame to be detected; obtains the safety limit of the rail operation based on the Sobel edge detection operator, the sliding window and the least squares method; calculates the perspective transformation matrix through ORB feature matching and SVD, and obtains the corresponding frame aligned with the frame to be detected through perspective transformation; obtains the obstacle detection result map based on the background subtraction method and the regional texture similarity method. Although this method combines the trajectory recognition and obstacle detection algorithms to realize an end-to-end trajectory obstacle detection and recognition system, the image gradient detection algorithm based on the Sobel edge detection operator has extremely poor robustness in complex environments and different lighting conditions and is unable to perform recognition tasks in actual scenarios. At the same time, the obstacle detection method based on background subtraction is similar to the template matching strategy and only relies on fixed prior information, which will lead to false detection and missed detection of intrusion targets. The overall method is not very practical.

[0004] The patent application with publication number CN108375771B discloses a detection method of a two-dimensional laser radar railway obstacle detection system, including an adjustment platform, a control operator and an alarm device are installed on the top left side of the adjustment platform, the control operator is located on the left side of the alarm device, and a two-dimensional laser radar is connected to the top center of the adjustment platform by screws. A detection method of a two-dimensional laser radar railway obstacle detection system, the detection method of the two-dimensional laser radar railway obstacle detection system has the following specific steps: first, the two-dimensional laser radar is adjusted to be parallel to the rail surface, and then the radar two-dimensional plane coordinate system is calibrated and the protection zone is set, and finally the obstacle judgment is confirmed. Compared with the obstacle detection method based on visual calculation, although this method can realize long-distance obstacle detection, it requires higher assembly accuracy and is not suitable for relatively bumpy trains. The calibration work in the early stage of the system will introduce calibration errors, and the real-time performance is not as good as the visual detection method.

[0005] Long-distance rail recognition and obstacle detection technology based on computer vision can effectively solve the problems of short visual distance and high false detection rate caused by the driver's visual inspection method, and low scanning frequency and insufficient real-time performance caused by lidar scanning imaging. At the same time, it reduces the impact of invalid target detection caused by mutual occlusion in multi-obstacle scenarios. Summary of the invention

[0006] In view of the defects of the prior art, the present invention provides a long-distance rail identification and obstacle detection method and system to solve the problems of the prior art such as limited application scenarios, low detection accuracy and poor real-time performance.

[0007] In order to achieve the above objectives, in a first aspect, the present invention provides a long-distance rail identification and obstacle detection method, comprising the following steps.

[0008] S1. Adjustment of optical system parameters and image acquisition: After the telephoto optical system is set up at the train driver's seat, the focus ring is controlled to achieve clear imaging of the rails 6 km away, and the camera exposure time is adjusted to ensure that the rails and nearby objects within the depth of field are within the visual range without overexposure. The high-speed motion camera acquisition function is turned on during the train's travel and the acquired images are transmitted to the back end in real time to perform upstream and downstream visual tasks.

[0009] S2. Image preprocessing: In an image, the semantic information of a railroad track is more special than other targets. It has two main characteristics: high gradient change rate and strong gradient continuity. Therefore, after the backend obtains the video frame captured by the camera, it first needs to preprocess the image to enhance the gradient information of the railroad track and reduce the gradient influence caused by background stray objects, including cropping the center area of ​​the image, grayscale conversion of the image, histogram equalization, and setting Gaussian blur to reduce the gradient information of the non-focused part.

[0010] Since the observation system is fixed in the train as a whole, it can remain relatively stationary with the train while it is running. When shooting at a long distance, the rails are always kept in the center of the frame without shifting due to the strong continuity of the rails and the small rotation of the curved rails. At the same time, considering that reducing the full-frame size can speed up frame processing and reduce the interference of other objects in space on rail identification, quantitative cropping is performed on the center area of ​​the full frame to reduce the resolution while ensuring the integrity of the rails.

[0011] In the actual rail recognition process, the main focus is on whether the image has the gradient information of the rail, so the color information of the image is not needed during processing. In order to reduce information redundancy and increase the number of real-time processing frames, the image needs to be gray-converted before the upstream recognition task. The gray-conversion of the image can reduce the number of channels while retaining texture and detail information.

[0012] In outdoor shooting, the ambient light directly affects the brightness of all objects in the field of view, so the actual visual experience is not good. The main feedback is that the brightness of all objects is similar, which weakens the gradient representation of the whole image. The contrast of the image can be significantly enhanced after the histogram equalization operation, which can highlight the light and dark alternating scenes with obvious gradient information and realize the visual enhancement of gradient detection. At the same time, in order to reduce the negative impact of other long objects like rails on rail recognition, a 5*5 pixel Gaussian blur kernel is used to convolve the entire image to reduce its overall gradient threshold.

[0013] S3. Railroad track recognition algorithm: After the image preprocessing operation is completed, the railroad track information in the picture needs to be extracted and identified. Ordinary edge detection operators (Sobel, Canny, etc.) are insufficient in edge information extraction capabilities in actual scenarios; on this basis, it is proposed to use a multi-dimensional gradient weighted average operator group as a convolution kernel to extract the gradient information of the image. The gradient has directionality in two-dimensional space, so 8 different directional gradient operators are set to perform convolution operations on the original image respectively, and 8 contour maps with different gradient features are obtained. After weighted averaging of these contour maps, a contour map with strong information of the railroad part corresponding to the original image is obtained.

[0014] The above contour map is input into the rail recognition algorithm. The rail recognition algorithm is based on some prior information. For the first frame of the image, the rail information on the left and right sides of the bottom edge of the image (the closest distance) needs to be annotated. After the annotation is completed, no other operations are required. Set two fixed-size left and right rail recognition frames, and use the prior information as the center point of the recognition frame. The algorithm slices each frame of the image from bottom to top, with a slice width of about 10-15 pixels; for each slice, the row is summed to obtain a one-dimensional matrix, and the two points with the highest and second highest peak values ​​are traversed in the one-dimensional matrix. By comparing the index values, it can be seen that the smaller index value is the point on the left rail, and the larger index value is the point on the right rail. The rail pixel point at the bottom edge is given as prior information. The rail pixel points are processed from bottom to top in sequence. After the pixel coordinates of the current slice are obtained, the horizontal pixel difference between the coordinates of the slice and the coordinates of the previous slice is calculated. The difference is used to move the horizontal position of the recognition frame, so as to meet the rail recognition requirements under the curved track. The width of the recognition frame gradually decreases with the increase in the number of slices, but it is always greater than the rail pixel width. After performing the same operation on all slices, the point sequence of the left and right tracks on the frame image can be obtained. The Bessel function curve is fitted to the point sequence to obtain the complete left and right rail curves under the current frame, completing the rail recognition task.

[0015] S4. Obstacle detection algorithm: In the target detection and recognition task of rail obstacles, we only need to care whether there are objects on the rails and their surrounding areas. Some objects far away from the rails are not the objects of concern. Therefore, after completing rail recognition, the region of interest (ROI) for target detection is delineated according to the pixel coordinates of the left and right rails. On the one hand, it can exclude the object information in the non-track area, and on the other hand, the ROI area is equivalent to cutting the original image, which speeds up the calculation efficiency of the detection model. The target detection and recognition model adopts the classic YOLOv5 detection algorithm, and performs corresponding training for specific scenes (railway track background) and specific obstacle targets (people, cars, animals, trees and stones, etc.) to obtain a detection model with higher adaptability to the current scene. Considering that in long-distance application scenarios, some movable obstacles may move toward the rails, the area within 2m on both sides of the rails and the left and right rails is set as the actual warning area. When the existence of the pre-trained target is detected and recognized in this area, the alarm device is triggered to remind the driver to perform corresponding operations.

[0016] In a second aspect, the present invention provides a long-distance rail identification and obstacle detection system design, comprising:

[0017] The high frame rate video shooting system based on the telephoto optical system is mainly composed of a front telephoto optical system and an ultra-high-speed industrial camera. Due to the large size and weight of the telephoto optical system, a frame module is required to support the ground to stabilize the system.

[0018] The high-performance computer platform is mainly characterized by high-speed data throughput and real-time image processing modules. The platform needs to receive 500 frames of images per second sent back by the shooting system and perform real-time processing and feedback to the front end.

[0019] Compared with the prior art, the present invention has the following beneficial effects:

[0020] (1) The present invention provides a long-distance rail identification and obstacle detection method and system. To address the shortcomings of the current distributed track detection method, such as large spatial sampling intervals and short identification distances due to fixed-position installation, which make it impossible to achieve full real-time requirements; and the track obstacle detection method based on laser radar, which has complex system devices and low sampling frequency, the present invention combines a telephoto optical system with a high-speed industrial camera to build a system, thereby achieving a high-frame rate detection method with strong real-time performance while ensuring the detection distance.

[0021] (2) The present invention provides a long-distance rail identification and obstacle detection method and system. A telephoto optical observation platform is arranged at the position of the train cockpit. The viewing angle needs to be directly facing the front rail to ensure that the rail is always in the center area of ​​the frame. Since the rail has the characteristics of continuity and small curvature of the curved rail, and the system field of view angle range can reach a spatial range of 15m*15m on a two-dimensional plane, there is no need to adjust the alignment and focus position of the optical system subsequently.

[0022] (3) The present invention provides a long-distance rail identification and obstacle detection method and system. The image feature extraction convolution operation based on the multi-dimensional gradient weighted average operator group can effectively extract the gradient information of the rail in complex scenes by using the continuity and slender features of the rail itself, and effectively weaken other objects with obvious gradients in the background layer, so as to obtain the rail features. According to the extracted feature map with rail gradient features, the image is segmented and sliced ​​along the y direction, and the slice at the lower edge is the initial position of the rail. The track is marked based on artificial prior information. This operation is only performed in the initial stage of system operation and does not require manual custody in the future. The manually marked position is the center position of the left and right track recognition frame. The recognition frame is used to query the pixel coordinates with the maximum gradient value of the left and right regions for the next slice (the upper layer of the slice at the lower edge of the image). At the same time, the width of the recognition frame decreases with a negative exponential function trend for each slice upward. When the pixel coordinates of the maximum gradient position obtained by the recognition frame deviate from the center coordinates of the recognition frame, the recognition frame will preferentially move the overall coordinates of the frame closer to the deviation direction before performing an operation on the next slice, so as to effectively handle special situations such as curved rails.

[0023] (4) The present invention provides a long-distance rail identification and obstacle detection method and system. After completing rail identification, the pixel coordinates obtained by identification are used to delineate the region of interest and perform obstacle detection. The target detection and recognition task uses the classic YOLOv5 detection model, which is retrained for actual rail scenes. Considering the category distribution of obstacles in actual scenes, the training objects are newly added special categories such as people, cars, animals, trees, and stones, which have a high probability of occupying and rushing the track. At the same time, considering the mobile targets that have not entered the track at the current time but may occupy the track in the future under long-distance detection, the system demarcates the area within 2m on both sides of the rail and the left and right tracks as the actual warning area, which broadens the background area for obstacle detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A schematic diagram of a hierarchical design of a system architecture provided in Example 1 of the present invention;

[0025] Figure 2 A schematic diagram of a high-speed video acquisition system based on a telephoto optical module provided in Embodiment 1 of the present invention;

[0026] Figure 3 A schematic diagram of an actual acquisition scenario provided in Example 1 of the present invention;

[0027] Figure 4 A schematic diagram of a flow chart of a rail identification algorithm provided in Embodiment 1 of the present invention;

[0028] Figure 5 A schematic diagram of a track obstacle detection and recognition model framework provided in Example 1 of the present invention; DETAILED DESCRIPTION

[0029] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the described embodiments are intended only to facilitate the understanding of the present invention and do not have any limiting effect on it. The accompanying drawings are in a very simplified form and use non-precise proportions, which are only used to conveniently and clearly assist in explaining the purpose of the embodiments of the present invention. The structures shown in the accompanying drawings are part of the actual structure. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0030] Embodiment 1,

[0031] A long-distance rail identification and obstacle detection method and system, suitable for long-distance rail obstacle detection and identification, the system architecture layered design diagram is as follows Figure 1 As shown, the following steps are included:

[0032] S1. Long-distance telephoto optical module high-speed video acquisition system layout as follows Figure 2As shown in the figure, considering the limited space in the cabin for the system, and the large size of the telephoto optical system due to its long focal length, the design adopts the Schmidt-Cassegrain reentrant optical system to reduce the system tube length while ensuring the unchanged focal length. The back end of the telephoto optical system is connected to a high-speed industrial camera, and the main part of the optical observation platform is fixed in the center of the cockpit through a mechanical structure. The system is connected to the computing platform through the CoaXPress protocol, thereby providing ultra-high bandwidth video streaming transmission.

[0033] S2. Actual acquisition scenarios of long-distance rail identification and obstacle detection systems Figure 3 As shown in the figure, the telephoto optical observation system faces the front of the rail. During installation, the direction and focus ring of the optical system need to be adjusted to ensure that the rail can be focused on several kilometers ahead. After the system is assembled and debugged, pre-processing and rail recognition operations are performed on each frame of the collected image. The rail recognition algorithm process is as follows Figure 4 As shown in the figure, a 1920*1080 pixel resolution image is input, and after the process of image preprocessing - image gradient information extraction - rail prior information labeling - left and right track recognition - Bessel function track fitting, an image with the original image as the background and a red rail line attached is finally obtained.

[0034] S3. Track obstacle detection and recognition model framework of long-distance rail recognition and obstacle detection system Figure 5 As shown in the figure, the most popular YOLOv5 target detection and recognition network in the detection field is used. The network consists of an input end, a backbone network, a neck network, and a head network. The input end performs Mosaic data enhancement on the input image, and performs adaptive image scaling and adaptive anchor frame calculation on the image to facilitate network training. The two important modules in the backbone network are the Focus structure and the CSP structure. The Focus module in YOLOv5 is to slice the image before the image enters the Backbone. The specific operation is to get a value for every other pixel in an image, which is similar to neighboring downsampling. In this way, four images are obtained. The four images are complementary but there is no information loss. In this way, the W and H information are concentrated in the channel space, and the input channel is expanded by 4 times, that is, the spliced ​​image becomes 12 channels relative to the original RGB three-channel mode. Finally, the new image is subjected to convolution operation, and finally a double down-sampling feature map without information loss is obtained. CSPNet mainly splits the feature map into two parts, one part is convolved, and the other part is concatenated with the result of the convolution operation of the previous part. The neck network adopts the CSP2 structure designed by CSPNet to enhance the network feature fusion capability. The head network uses CIOU_LOSS as the loss function of the bounding box to predict the target position.

[0035] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A long-distance rail identification and obstacle detection method and system, characterized in that: The steps include: 1) After the telephoto optical system is set up at the train driver's seat, the focus ring is controlled to achieve clear imaging of the rails 6 km away, and the camera exposure time is adjusted to ensure that the rails and nearby objects within the depth of field are within the visual range without overexposure. During the train's travel, the high-speed motion camera acquisition function is turned on and the acquired images are transmitted to the back end in real time to perform upstream and downstream visual tasks. 2) After the backend obtains the video frame captured by the camera, it first needs to preprocess the image to enhance the gradient information of the railroad track and reduce the gradient impact caused by background stray objects, including cropping the center area of ​​the image, image grayscale conversion, histogram equalization, and setting Gaussian blur to reduce the gradient information of the non-focus part. 3) After the image preprocessing operation is completed, the rail information in the picture needs to be extracted and identified. Ordinary edge detection operators (Sobel, Canny, etc.) are insufficient in edge information extraction capabilities in actual scenarios; on this basis, it is proposed to use a multi-dimensional gradient weighted average operator group as a convolution kernel to extract the gradient information of the image. The gradient has directionality in two-dimensional space, so 8 different directional gradient operators are set to perform convolution operations on the original image respectively, and 8 contour maps with different gradient features are obtained. After weighted averaging of these contour maps, a contour map with strong information of the rail part corresponding to the original image is obtained. The contour map is input into the rail recognition algorithm. The rail recognition algorithm is based on some prior information. For the first frame of the image, the rail information on the left and right sides of the bottom edge of the image (the closest distance) needs to be annotated. After the annotation is completed, no other operations are required. Set two fixed-size left and right rail recognition frames, and use the prior information as the center point of the recognition frame. The algorithm slices each frame of the image from bottom to top, and the slice width is about 10-15 pixels; For each slice, the row sum is taken to obtain a one-dimensional matrix. The two points with the highest and second highest peak values ​​are searched in the one-dimensional matrix. By comparing the index values, it can be seen that the point with the smaller index value is on the left rail, and the point with the larger index value is on the right rail. The rail pixel point at the bottom edge is given as prior information. The rail pixel points are processed from bottom to top in sequence. After the pixel coordinates of the current slice are obtained, the horizontal pixel difference between the coordinates of the slice and the coordinates of the previous slice is calculated. The difference is used to move the horizontal position of the recognition frame, so as to meet the rail recognition requirements under the curved rail. The width of the recognition frame gradually decreases with the increase in the number of slices, but it is always greater than the rail pixel width. After performing the same operation on all slices, the point column of the left and right rails on the frame image can be obtained. The Bessel function curve is fitted to the point column to obtain the complete left and right rail curves under the current frame, and the rail recognition task is completed. 4) The region of interest (ROI) for target detection is defined based on the pixel coordinates of the left and right rails. On the one hand, it can exclude object information in non-track areas. On the other hand, the ROI area is equivalent to cropping the original image, which speeds up the calculation efficiency of the detection model. The target detection and recognition model uses the classic YOLOv5 detection algorithm, and performs corresponding training for specific scenes (railway track background) and specific obstacle targets (people, cars, animals, trees, stones, etc.) to obtain a detection model with higher adaptability to the current scene. Considering that in long-distance application scenarios, some movable obstacles may move toward the rails, the area within 2m on both sides of the rails and the left and right tracks is set as the actual warning area. When the presence of the pre-trained target is detected and identified in this area, the alarm device is triggered to remind the driver to perform corresponding operations.

2. A long-distance rail identification and obstacle detection method and system as claimed in claim 1, characterized in that: In order to solve the problems of short detection distance, complex system installation and low sampling frequency in the existing methods, a telephoto optical system is combined with a high-speed industrial camera to build an optical observation system platform. While ensuring the detection distance, a high-frame rate detection method with strong real-time performance is achieved.

3. A long-distance rail identification and obstacle detection method and system as claimed in claim 1, characterized in that: A telephoto optical observation platform is arranged at the position of the train cockpit, and the viewing angle needs to be facing the front rail to ensure that the rail is always in the center of the frame. Since the rail has the characteristics of continuity and small curvature of the curved rail, and the system field of view angle range can reach 15m*15m in the two-dimensional plane, there is no need to adjust the alignment and focus position of the optical system later.

4. A long-distance rail identification and obstacle detection method and system as claimed in claim 1, characterized in that: The image feature extraction convolution operation based on the multi-dimensional gradient weighted average operator group can effectively extract the gradient information of the rail in complex scenes by using the continuity and slender features of the rail itself, and effectively weaken other objects with obvious gradients in the background layer, so as to obtain the rail features. According to the extracted feature map with rail gradient features, the image is segmented along the y direction, and the slice at the lower edge is the initial position of the rail. The track is marked based on artificial prior information. This operation is only performed in the initial stage of system operation and does not require manual hosting in the future. The manually marked position is the center position of the left and right track recognition frame. The recognition frame is used to query the pixel coordinates with the maximum gradient value in the left and right regions for the next slice (the previous layer of the slice at the lower edge of the image). At the same time, the width of the recognition frame decreases with a negative exponential function trend for each slice upward. When the pixel coordinates of the maximum gradient position obtained by the recognition frame deviate from the center coordinates of the recognition frame, the recognition frame will preferentially move the overall coordinates of the frame closer to the deviation direction before performing operations on the next slice, which can effectively handle special cases such as curved rails.

5. A long-distance rail identification and obstacle detection method and system as claimed in claim 1, characterized in that: The target detection and recognition tasks use the classic YOLOv5 detection model, which is retrained for actual rail scenes. Considering the category distribution of obstacles in actual scenes, the training objects are new special categories such as people, cars, animals, trees and stones, which have a high probability of occupying and rushing the track. At the same time, considering the mobile targets that have not entered the track at the current time but may occupy the track in the future under long-distance detection, the system defines the area within 2m on both sides of the rails and the left and right tracks as the actual warning area, which broadens the background area for obstacle detection.

Images