Railway unmanned obstacle detection and positioning method, device, equipment and medium

Abstract

The present invention relates to the field of assisted driving technology and discloses a method, device, equipment, and medium for detecting and locating obstacles in unmanned railways. The method comprises: clustering point cloud data of unknown obstacles, extracting features from the clustering results, and obtaining the position and size of the unknown obstacles; voxelizing the point cloud data of known obstacles, and extracting multi-scale features of the known obstacles in the voxelized cylinders using a convolutional neural network to obtain the position and size of the known obstacles; inputting a track image into a deep convolutional network to extract multi-scale features of the track image and segment the track line; and determining the travel track. Based on the positional relationship between the positions of the unknown and known obstacles and the travel track, as well as the obstacle size, determining whether there is an obstacle on the travel track. The present invention is applicable to rail transportation in specific scenarios such as railways and mines, ensuring safety during transportation and improving transportation efficiency.

Description

Technical Field

[0001] The present invention relates to the field of assisted driving technology, and in particular to a method, device, equipment and medium for detecting and positioning obstacles in unmanned railway driving. Background Art

[0002] The research, development and operation of unmanned locomotive systems can help reduce the probability of transportation accidents caused by scheduling and operational errors. In addition, the unmanned vehicle system can operate 24 hours a day, which can improve transportation efficiency while reducing labor costs.

[0003] An important prerequisite for the safe operation of unmanned locomotive systems is that the locomotive can autonomously identify various obstacles in front of it, including pedestrians, cars, and obstacles of unknown categories. The existing method of identifying obstacles is to use models trained based on scene data. Its advantage is that it can learn the characteristics of various common obstacles and provide accurate obstacle category information during model inference. However, unlike rail transit, industrial transport railways are not closed and their surroundings are more complex. If a model-based obstacle detection method is used, it is easy to miss uncommon obstacles.

[0004] Image-based obstacle detection has the advantages of low cost and rich semantic information. However, images are easily affected by lighting conditions and bad weather, have poor stability, and cannot provide accurate obstacle distance information, posing a great challenge to camera single-mode recognition.

[0005] The obstacle detection method based on lidar can accurately measure the distance from the obstacle to the locomotive, but the radar point cloud data has fewer features and the data annotation cost is high.

[0006] The combination of cameras and lidar can combine the advantages of both to obtain accurate obstacle status information, including obstacle coordinates, the distance of the obstacle from the locomotive, obstacle category information and track area, but it is still difficult to distinguish between moving tracks and non-moving tracks in the switch area.

[0007] The route map is the locomotive route obtained based on the locomotive dispatch plan and actual scene mapping. The route is obtained by combining the locomotive's positioning information and matched with the track area to distinguish between traveling tracks and non-traveling tracks. Summary of the Invention

[0008] The present invention provides a method, device, equipment and medium for detecting and locating obstacles in an unmanned railway system. Combined with a route map, the advantages of cameras and lidar can be fully utilized to solve the above-mentioned problems.

[0009] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0010] In a first aspect, the present invention provides a method for detecting and locating obstacles in an unmanned railway system, comprising:

[0011] Jointly calibrate the lidar and camera so that the point cloud data collected by the lidar and the track image collected by the camera correspond to each other in the same coordinate system;

[0012] Retain the point cloud data of unknown obstacles in the point cloud data and cluster them, perform feature extraction on the clustering results, and obtain the position and size of the unknown obstacles;

[0013] The point cloud data of known obstacles is voxelized using the PointPillars network to obtain voxelized cylinders. The multi-scale features of the known obstacles in the voxelized cylinders are extracted using a convolutional neural network. The multi-scale features of the known obstacles are aggregated into single-scale features to obtain the location and size of the known obstacles.

[0014] Input the track image into the deep convolutional network to extract the multi-scale features of the track image and realize the segmentation of the track line;

[0015] The positions of unknown obstacles and known obstacles are mapped to the image coordinate system. Combined with the track segmentation results, the route map is matched with the track segmentation results to determine the travel track.

[0016] Whether there is an obstacle on the track is determined based on the position of the unknown obstacle mapped to the image coordinate system, the position relationship between the position of the known obstacle and the track, and the sizes of the unknown obstacle and the known obstacle.

[0017] In one embodiment, retaining the point cloud data of unknown obstacles in the point cloud data and clustering them, performing feature extraction on the clustering results, and obtaining the position and size of the unknown obstacles specifically include:

[0018] Remove noise points and ground point clouds from the point cloud data collected by the lidar, retain the point cloud data representing obstacles, traverse each point in the point cloud data, take a point p as the starting point, find all points whose Euclidean distance to point p is less than a threshold and classify them into the same temporary cluster C; then repeat the Euclidean distance-based clustering process for each point in the temporary cluster C, continuously expanding the range of the temporary cluster C until no new points are added to the temporary cluster C; each temporary cluster C represents a potential unknown obstacle, and the center position of the temporary cluster C is used as the position of the unknown obstacle, and the maximum difference between the points of the temporary cluster C on each coordinate axis is used as the size of the unknown obstacle.

[0019] In one embodiment, the step of inputting the track image into a deep convolutional network to extract multi-scale features of the track image and segment the track line specifically includes:

[0020] The deep convolutional network adopts the YOLOv8-seg network, segments the track lines in the track image through mask branches, and uses morphological operations to denoise the track line segmentation results.

[0021] In a second aspect, the present invention provides a railway unmanned obstacle detection and positioning device, comprising:

[0022] The calibration module jointly calibrates the lidar and camera so that the point cloud data collected by the lidar and the track image collected by the camera correspond to each other in the same coordinate system;

[0023] The unknown obstacle detection module retains the point cloud data of unknown obstacles in the point cloud data and clusters them, extracts features from the clustering results, and obtains the position and size of the unknown obstacles;

[0024] The known obstacle detection module voxelizes the point cloud data of known obstacles using the PointPillars network to obtain voxelized cylinders. The convolutional neural network is used to extract the multi-scale features of the known obstacles in the voxelized cylinders, and the multi-scale features of the known obstacles are aggregated into single-scale features to obtain the location and size of the known obstacles.

[0025] Track line segmentation module: inputs the track image into the deep convolutional network, extracts the multi-scale features of the track image, and realizes track line segmentation;

[0026] The travel track matching module maps the positions of unknown obstacles and known obstacles to the image coordinate system, combines the track line segmentation results, matches the travel route map with the track line segmentation results, and determines the travel track;

[0027] The track obstacle detection module determines whether there is an obstacle on the track based on the position of the unknown obstacle mapped to the image coordinate system, the position relationship between the position of the known obstacle and the track, and the sizes of the unknown obstacle and the known obstacle.

[0028] In one embodiment, in the unknown obstacle detection module, retaining the point cloud data of unknown obstacles in the point cloud data and clustering them, performing feature extraction on the clustering results, and obtaining the position and size of the unknown obstacles specifically include:

[0029] Remove noise points and ground point clouds from the point cloud data collected by the lidar, retain the point cloud data representing obstacles, traverse each point in the point cloud data, take a point p as the starting point, find all points whose Euclidean distance to point p is less than a threshold and classify them into the same temporary cluster C; then repeat the Euclidean distance-based clustering process for each point in the temporary cluster C, continuously expanding the range of the temporary cluster C until no new points are added to the temporary cluster C; each temporary cluster C represents a potential unknown obstacle, and the center position of the temporary cluster C is used as the position of the unknown obstacle, and the maximum difference between the points of the temporary cluster C on each coordinate axis is used as the size of the unknown obstacle.

[0030] In one embodiment, in the track line segmentation module, the track image is input into a deep convolutional network to extract multi-scale features of the track image to achieve track line segmentation, which specifically includes:

[0031] The deep convolutional network adopts the YOLOv8-seg network, segments the track lines in the track image through mask branches, and uses morphological operations to denoise the track line segmentation results.

[0032] In a third aspect, the present invention provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and wherein the processor implements the steps of the method of any one of the embodiments of the first aspect when executing the computer program.

[0033] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, characterized in that when the computer program is executed by a processor, the steps of the method of any one of the embodiments in the first aspect are implemented.

[0034] Compared with the prior art, the beneficial technical effects of the present invention are:

[0035] (1) The present invention can ensure the safe operation of vehicles or equipment in complex environments, especially in scenarios where unknown obstacles exist and track lines need to be accurately identified.

[0036] (2) The present invention is applicable to rail transportation in specific scenarios such as railways and mines, thereby ensuring safety during transportation, improving transportation efficiency, and reducing manual intervention. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 4 is a flow chart of a method in an embodiment of the present invention.

[0038] Figure 2 Schematic diagram of the system used in the embodiment of the present invention.

[0039] Figure 3Schematic diagram of an obstacle on a non-travel track during normal operation of a locomotive in an embodiment of the present invention.

[0040] Figure 4 This is a schematic diagram of an obstacle on the traveling track during normal operation of a locomotive in an embodiment of the present invention.

[0041] Figure 5 This is a schematic diagram of a locomotive in an embodiment of the present invention operating normally with no obstacle on the track. DETAILED DESCRIPTION

[0042] A preferred embodiment of the present invention will be described in detail below with reference to the accompanying drawings.

[0043] like Figure 2 As shown in the figure, the present invention provides a method for detecting and locating obstacles in unmanned railway systems. The detection system employed includes a vehicle-mounted laser radar (LIDAR) and cameras (which may include both near-focus and far-focus cameras). These cameras are connected to a processor via Ethernet, and detection results are directly transmitted to the vehicle controller. The detection system can be applied to locomotives, such as industrial railway locomotives, to detect obstacles in switch areas. Industrial railway locomotives can be categorized as either unmanned (driverless) or manned (driverless).

[0044] The following will take the obstacle detection and positioning of the UAV when running in the switch area as an example to illustrate the technical solution of the present invention.

[0045] like Figure 1 As shown, a method for detecting and locating obstacles in an unmanned railway system according to the present invention comprises the following steps:

[0046] S1: Jointly calibrate the lidar and camera so that the point cloud data collected by the lidar and the track image collected by the camera correspond to each other in the same coordinate system;

[0047] S2: retain the point cloud data of unknown obstacles in the point cloud data and cluster them, perform feature extraction on the clustering results, and obtain the position and size of the unknown obstacles;

[0048] S3: The point cloud data of known obstacles is voxelized using the PointPillars network to obtain voxelized cylinders. The multi-scale features of the known obstacles in the voxelized cylinders are extracted using a convolutional neural network. The multi-scale features of the known obstacles are aggregated into single-scale features to obtain the location and size of the known obstacles.

[0049] S4: Input the track image into the deep convolutional network to extract the multi-scale features of the track image and realize the segmentation of the track line;

[0050] S5: Map the positions of unknown obstacles and known obstacles to the image coordinate system, combine the track segmentation results, match the route map with the track segmentation results, and determine the travel track;

[0051] S6: Determine whether there is an obstacle on the travel track according to the position of the unknown obstacle mapped to the image coordinate system, the position relationship between the position of the known obstacle and the travel track, and the sizes of the unknown obstacle and the known obstacle.

[0052] Specifically, in step S1, the lidar and camera are jointly calibrated. This is the foundation of the entire detection system and is crucial for the accuracy of subsequent multi-sensor data fusion. Lidar accurately measures the three-dimensional spatial position and distance of target objects, while cameras capture rich visual images, including semantic features such as texture and color. To ensure accurate correspondence between the two data sets in the same coordinate system, the lidar's external parameters and the camera's internal parameters must be calibrated.

[0053] The extrinsic parameters of a lidar primarily consist of a rotation matrix and a translation matrix relative to the camera and the platform (such as a vehicle or robot). The rotation matrix describes the rotation of the lidar coordinate system relative to other coordinate systems, while the translation matrix represents its positional offset. By accurately calculating these two matrices, the point cloud data captured by the lidar can be accurately mapped to the camera's image coordinate system.

[0054] Internal camera parameters, such as the coordinates of the camera's principal light spot and focal length, have a significant impact on image quality and accuracy. The principal light spot coordinates determine the image's center position, while the focal length determines the image's scale. Calibration of these parameters eliminates image distortion and improves image data quality. After calibration, the lidar uses raw point cloud data and the camera uses raw image data, preserving the most complete information and providing a rich data foundation for subsequent processing.

[0055] In one embodiment, step S2 of retaining the point cloud data of unknown obstacles in the point cloud data and clustering them, performing feature extraction on the clustering results, and obtaining the position and size of the unknown obstacles specifically includes:

[0056] Remove noise points and ground point clouds from the point cloud data collected by the lidar, retain the point cloud data representing obstacles, traverse each point in the point cloud data, take a point p as the starting point, find all points whose Euclidean distance to point p is less than a threshold and classify them into the same temporary cluster C; then repeat the Euclidean distance-based clustering process for each point in the temporary cluster C, continuously expanding the range of the temporary cluster C until no new points are added to the temporary cluster C; each temporary cluster C represents a potential unknown obstacle, and the center position of the temporary cluster C is used as the position of the unknown obstacle, and the maximum difference between the points of the temporary cluster C on each coordinate axis is used as the size of the unknown obstacle.

[0057] Specifically, the present invention obtains unknown obstacle information using the Euclidean clustering algorithm in the following manner.

[0058] 1) After using LiDAR to collect point cloud data, since there is a lot of noise in the original point cloud data, such as ground point clouds, stray reflection points, etc., which will interfere with subsequent obstacle detection, it is necessary to pre-process the point cloud data collected by the LiDAR. Use statistical filtering, radius filtering and other methods to filter out noise points, remove irrelevant information such as noise points and ground point clouds, and retain point cloud data that may represent obstacles. The Euclidean clustering algorithm performs clustering based on the Euclidean distance between points. The Euclidean distance refers to the straight-line distance between two points in Euclidean space. For any two points in the point cloud data, by calculating the Euclidean distance between them, when the distance is less than the set threshold, the two points are considered to belong to the same cluster: for any two points in the point cloud and , the Euclidean distance between them The calculation formula is:

[0059] ;

[0060] When the Euclidean distance between two points is less than the set threshold , the two points are considered to belong to the same cluster.

[0061] 2) Use the Euclidean clustering algorithm to divide the processed point cloud data into different clusters. The algorithm will traverse each point in the point cloud, starting from a point p, and find all points with a Euclidean distance less than a threshold. points, and classify them into the same temporary cluster C. Then repeat the above process for each point in the temporary cluster C, continuously expanding the range of the cluster until no new points can be added to the cluster. Each cluster finally formed represents a potential unknown obstacle.

[0062] 3) Extract features from each cluster, including calculating the cluster's center position, size, point cloud density and other features, so as to describe and analyze unknown obstacles later. , Cluster The i-th point in cluster Central location The calculation formula is:

[0063] ; ; ;

[0064] yes The x-axis coordinate, y-axis coordinate, and z-axis coordinate of the cluster; the size of the cluster can be determined by calculating the maximum difference of each point in the cluster on each coordinate axis, and the point cloud density can be calculated by the ratio of the number of points n in the cluster to the spatial volume V occupied by the cluster.

[0065] Specifically, the clustering implementation process of point cloud data includes:

[0066] Establishing a search structure: To improve search efficiency, a KD-tree (K-Dimensional Tree, KD-Tree) data structure is often used to organize point cloud data. A KD-tree is a binary search tree that recursively divides the point cloud space into distinct regions, allowing for a quicker search when searching for neighboring points of a given point.

[0067] Select starting point: Randomly select an unvisited point from the point cloud as the starting point.

[0068] Neighborhood search: With the starting point as the center, search the KD tree for all nodes whose Euclidean distance to it is less than the set threshold. These points constitute the neighborhood of the starting point. The choice of is very critical, it determines the tightness of the clustering. If it is too large, different obstacles may be merged into one cluster; If it is too small, an obstacle may be split into multiple clusters.

[0069] Expand the cluster: mark the starting point and the points in its neighborhood as belonging to the same temporary cluster C, and repeat the above neighborhood search and marking process for each point in the neighborhood, continuously expanding the range of the cluster until no new points can be added to the cluster.

[0070] Repeat the above steps: Continue to select unvisited points as new starting points, and repeat the above process until all points in the point cloud have been visited. Eventually, each formed cluster represents a potential unknown obstacle.

[0071] In one embodiment, in step S3, known obstacles are detected by the following steps:

[0072] S31, extracting key point cloud data containing obstacles from the point cloud data, and screening and intercepting the key point cloud data as sample point clouds for establishing a dataset for obstacle detection.

[0073] S32, perform data enhancement processing on the obtained sample point cloud, perform data annotation on the obstacle point cloud, including feature information such as obstacle category, 3D target frame, direction, etc., store the obstacle point cloud and annotation information as a dataset in KITTI format, and divide the obtained dataset into a training set and a test set.

[0074] S33, builds an obstacle detection model and formulates its data configuration file and model configuration file based on the analysis results of the collected point cloud data. The obstacle detection model is based on a feature extraction network constructed using the PointPillars network. As a 3D target detection model, the PointPillars network can process 3D point clouds into 2D pseudo images, and then use traditional convolutional neural networks to extract features from the pseudo images, greatly improving the processing speed of point cloud data.

[0075] S34, training the obstacle detection model. After the training is completed, the obstacle point cloud to be detected is input into the trained obstacle detection model for detection, thereby completing the detection and identification of known obstacles in the industrial railway scenario.

[0076] The point cloud of obstacles to be detected is input into the trained obstacle detection model for detection, completing the detection and identification of known obstacles in the industrial railway scenario. The specific steps include:

[0077] S341 first performs voxelization, dividing the 3D point cloud data into vertical voxelized pillars (pillars), each containing a certain number of points. This is done to convert the 3D point cloud data into a 2D representation for easier subsequent processing.

[0078] S342, then use the PointPillars network to perform feature extraction and obstacle detection. The structure of the PointPillars network mainly includes three parts: voxelized cylindrical feature encoding, backbone network and detection head.

[0079] S343, Voxelized Cylinder Feature Encoding: For each point within the voxelized cylinder, its offset from the cylinder center and its relative position within the cylinder are calculated. These features are then combined with the original coordinates and reflection intensity of the point to obtain an enhanced feature for each point. Next, a maximum pooling operation is performed on the point features within each voxelized cylinder to obtain a feature representation for each voxelized cylinder.

[0080] S344, Backbone Network: Uses a two-dimensional convolutional neural network to process the voxelized pillars. It typically adopts a structure similar to a residual network (ResNet), consisting of multiple convolutional layers and residual blocks. These convolutional layers and residual blocks can learn different features of the point cloud data, such as object shape and edges. By performing convolution operations at different scales, the network can extract multi-scale features. For example, a small-scale convolution kernel can capture detailed information about obstacles, while a large-scale convolution kernel can grasp the overall distribution of obstacles. Finally, the extracted multi-scale features are aggregated and converted into single-scale features for subsequent accurate detection and location of known obstacles.

[0081] S345: Detection Head: Based on the extracted single-scale features, the detection head is responsible for predicting the 3D position, size, orientation, and category of known obstacles in space. Anchor-based detection is typically used, with anchor boxes of different scales and proportions predefined. Regression and classification operations are then used to determine the obstacle information corresponding to each anchor box.

[0082] In one embodiment, in step S4, the track image is input into a deep convolutional network to extract multi-scale features of the track image to achieve track line segmentation, which specifically includes:

[0083] The deep convolutional network adopts the YOLOv8-seg network, segments the track lines in the track image through mask branches, and uses morphological operations to denoise the track line segmentation results.

[0084] The technical solution of step S4 is introduced in detail below.

[0085] Track images are captured by a camera and fed into a deep convolutional network based on the YOLOv8-seg network for processing. The YOLOv8-seg network is a single-stage image segmentation algorithm, improved upon by adding a mask branch to the detection head. During training, the YOLOv8-seg network simultaneously optimizes detection and segmentation tasks, using a joint loss function to ensure both tasks are improved. During inference, the YOLOv8-seg network outputs the bounding box of the target and a pixel-level mask of the corresponding area, enabling instant object detection and segmentation. The overall architecture of the YOLOv8-seg network consists of three components: the backbone network, the neck network, and the detection head.

[0086] Backbone network: This network is used to extract features from track images. This feature extraction process typically involves a decrease in feature map resolution and an increase in the number of channels. The track image resolution is gradually downsampled from 640x640 to 20x20, and the number of input channels increases from 3 to 512. The backbone network primarily consists of the CBS module, the C2f module, and the SPPF module.

[0087] The CBS module represents a convolutional group consisting of a convolutional layer (Conv), a batch normalization layer (BN), and a SILU function. The backbone network contains five CBS modules, resulting in five downsampling cycles. The C2f module is designed for deep image feature extraction. While its output feature map maintains the same size as its input feature map, it has multiple internal branches. This structure splits the feature map along the channel dimension, with one portion passing through the convolutional layer for feature extraction, while the other portion skips the convolutional layer. Finally, the two feature maps are concatenated. This structure reduces network computation while enhancing feature representation. The SPPF module consists of pooling operations at different scales, which concatenate feature maps of different scales to improve detection of objects of varying sizes. Through convolutional and pooling layers of varying scales, the backbone network extracts multi-scale features from the image, providing rich information for subsequent track segmentation.

[0088] Neck Network: Using a Path Aggregation Network (PANet) architecture, it fuses features through bottom-up and top-down pathways. The bottom-up pathway conveys low-level feature information rich in details, while the top-down pathway conveys high-level feature information with strong semantic information. In this way, feature information at different scales complements each other, improving feature expressiveness and enhancing track recognition.

[0089] Detection Head: This head utilizes a decoupled design, separating the classification and regression tasks and integrating the detection and segmentation branches. The classification task determines whether each region in the image belongs to a track line, while the regression task predicts the position and shape of the track line. This decoupled design reduces interference between tasks and improves detection accuracy. In the track line segmentation task, the feature map output by the neck network is combined with the mask branch to predict and segment the track lines in the image. By analyzing image features, the position and shape of the track lines in the image are identified, generating the track line segmentation result.

[0090] Finally, the segmentation results are post-processed, such as using morphological operations (dilation, erosion, etc.) to remove noise and mis-segmented parts, and using connected component analysis to merge or remove small segmented areas to improve the accuracy and reliability of track segmentation.

[0091] In one embodiment, the steps of S5 include mapping the positions of unknown obstacles and known obstacles to the image coordinate system, combining the track segmentation results, matching the route map with the track segmentation results, and determining the travel track, which specifically includes:

[0092] The positions of the unknown obstacles detected in step S2 and the known obstacles detected in step S3 are mapped from the LiDAR coordinate system to the image coordinate system. This requires using the rotation matrix and translation matrix obtained in step 1 to convert the 3D point cloud coordinates in the LiDAR coordinate system into 2D pixel coordinates in the image coordinate system using the coordinate transformation formula.

[0093] After ensuring that the obstacle information and image information are in the same coordinate system, the route map is then matched with the track segmentation results in combination with the track segmentation results in the image. Specifically, a feature matching algorithm, such as one based on keypoint matching, can be used. First, feature points, such as corner points and edge points, are extracted from both the route map and the track segmentation results. Descriptors (such as SIFT and SURF) are then used to describe the characteristics of these feature points. Finally, by comparing the feature point descriptors, corresponding feature points in the route map and the track segmentation results are identified, thereby determining the location of the track.

[0094] In one embodiment, step S6 of determining whether there is an obstacle on the travel track based on the position of the unknown obstacle mapped to the image coordinate system, the positional relationship between the positions of the known obstacles and the travel track, and the sizes of the unknown obstacle and the known obstacles specifically includes:

[0095] Analyze the relationship between the position of the obstacle mapped to the image and the travel track to determine whether there is an obstacle on the travel track.

[0096] It can also include: if there is an obstacle, further determine the type (unknown or known), location and size of the obstacle. According to the situation of the obstacle, it provides a basis for subsequent decision-making and control.

[0097] For example, if an obstacle is close to the locomotive and on the track, braking or evasive action may be necessary. By calculating the relative distance and speed between the obstacle and the locomotive, the time to collision can be predicted, and the appropriate action can be determined based on the time to collision. If the time to collision is short, braking and stopping are necessary; if the time to collision is long, evasive action, such as honking the horn and slowing down, can be taken.

[0098] If the obstacle is far away or not on the track, you can continue driving normally, but you need to remain vigilant. You can set a certain safety distance threshold and warning area. When the obstacle enters the warning area, a warning signal will be issued to remind the driver or control system to pay attention to the obstacle.

[0099] As an example, Figure 3 、 Figure 4 and Figure 5 The following are schematic diagrams of the locomotive in normal operation, with the obstacle in front of it being on the non-travel track, on the travel track, and not on the track. For the first case, if Figure 3 As shown, track A is a non-travel track and track B is a travel track. At this time, the obstacle is on the non-travel track and does not affect the driving safety of the locomotive. In this case, braking is not required, but early warning is required. For the second case, Figure 4 As shown, the obstacle is on the track and the vehicle needs to be safely braked to stop. For the third case, Figure 5 As shown, the obstacle is not on the track and does not affect the driving safety of the locomotive. In this case, braking is not required, but early warning is required.

[0100] In summary, the present invention provides a method for detecting and locating obstacles on unmanned railways. This method uses laser radar point cloud data and track image data, combined with a route map, to make a fusion judgment and control the operating status of the locomotive. This method can be applied to unmanned obstacle detection on open industrial railways, ensuring locomotive safety during transportation and improving overall transportation efficiency.

[0101] It should be understood that although the steps in the flowcharts of the accompanying drawings are shown in sequence as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the accompanying drawings may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times. The order of execution of these steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a portion of the steps or stages in other steps.

[0102] Based on the description of the above method embodiments, the present invention also provides a device. The device may be a system (including a distributed system), software (application), module, component, server, client, etc. that uses the method described in the embodiments of this specification and is combined with the necessary implementation hardware. Based on the same innovative concept, the device in one or more embodiments provided in the embodiments of the present disclosure is as described in the following embodiments. Since the implementation scheme of the device to solve the problem is similar to the method, the implementation of the specific device in the embodiments of this specification can refer to the implementation of the aforementioned method, and the repetitions will not be repeated. As used below, the term "module" or "module group" refers to a combination of software and / or hardware that can implement a predetermined function. Although the device described in the following embodiments is preferably implemented in software, implementation in hardware, or a combination of software and hardware, is also possible and conceived.

[0103] A railway unmanned obstacle detection and positioning device, comprising:

[0104] The calibration module jointly calibrates the lidar and camera so that the point cloud data collected by the lidar and the track image collected by the camera correspond to each other in the same coordinate system;

[0105] The unknown obstacle detection module retains the point cloud data of unknown obstacles in the point cloud data and clusters them, extracts features from the clustering results, and obtains the position and size of the unknown obstacles;

[0106] The known obstacle detection module voxelizes the point cloud data of known obstacles using the PointPillars network to obtain voxelized cylinders. The convolutional neural network is used to extract the multi-scale features of the known obstacles in the voxelized cylinders, and the multi-scale features of the known obstacles are aggregated into single-scale features to obtain the location and size of the known obstacles.

[0107] Track line segmentation module: inputs the track image into the deep convolutional network, extracts the multi-scale features of the track image, and realizes track line segmentation;

[0108] The travel track matching module maps the positions of unknown obstacles and known obstacles to the image coordinate system, combines the track line segmentation results, matches the travel route map with the track line segmentation results, and determines the travel track;

[0109] The track obstacle detection module determines whether there is an obstacle on the track based on the position of the unknown obstacle mapped to the image coordinate system, the position relationship between the position of the known obstacle and the track, and the sizes of the unknown obstacle and the known obstacle.

[0110] In one embodiment, in the unknown obstacle detection module, retaining the point cloud data of unknown obstacles in the point cloud data and clustering them, performing feature extraction on the clustering results, and obtaining the position and size of the unknown obstacles specifically include:

[0111] Remove noise points and ground point clouds from the point cloud data collected by the lidar, retain the point cloud data representing obstacles, traverse each point in the point cloud data, take a point p as the starting point, find all points whose Euclidean distance to point p is less than a threshold and classify them into the same temporary cluster C; then repeat the Euclidean distance-based clustering process for each point in the temporary cluster C, continuously expanding the range of the temporary cluster C until no new points are added to the temporary cluster C; each temporary cluster C represents a potential unknown obstacle, and the center position of the temporary cluster C is used as the position of the unknown obstacle, and the maximum difference between the points of the temporary cluster C on each coordinate axis is used as the size of the unknown obstacle.

[0112] In one embodiment, in the track line segmentation module, the track image is input into a deep convolutional network to extract multi-scale features of the track image to achieve track line segmentation, which specifically includes:

[0113] The deep convolutional network adopts the YOLOv8-seg network, segments the track lines in the track image through mask branches, and uses morphological operations to denoise the track line segmentation results.

[0114] In one embodiment, the present invention provides a computer device, which may be a server. The computer device includes a processor, a memory, and a network interface connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store data used in the above method. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, the above method is implemented.

[0115] In one embodiment, the present invention further provides a computer-readable storage medium, such as a memory, comprising instructions, wherein the instructions are executable by a processor to perform the above method. The storage medium may be a computer-readable storage medium, such as a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, or optical data storage device.

[0116] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0117] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as illustrative and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. It is intended that all variations within the meaning and range of equivalents of the claims be embraced herein, and any reference signs in the claims should not be construed as limiting the claims to which they relate.

Claims

1. A method for detecting and locating obstacles in an unmanned railway system, characterized in that: include: Jointly calibrate the lidar and camera so that the point cloud data collected by the lidar and the track image collected by the camera correspond to each other in the same coordinate system; The point cloud data of unknown obstacles in the point cloud data is retained and clustered. Feature extraction is performed on the clustering results to obtain the position and size of the unknown obstacles. Each temporary cluster C obtained by clustering represents a potential unknown obstacle. The center position of the temporary cluster C is used as the position of the unknown obstacle, and the maximum difference between the points of the temporary cluster C on each coordinate axis is used as the size of the unknown obstacle. The point cloud data of known obstacles is voxelized using the PointPillars network to obtain voxelized cylinders. The multi-scale features of the known obstacles in the voxelized cylinders are extracted using a convolutional neural network. The multi-scale features of the known obstacles are aggregated into single-scale features to obtain the location and size of the known obstacles. Input the track image into the deep convolutional network to extract the multi-scale features of the track image and realize the segmentation of the track line; The positions of unknown obstacles and known obstacles are mapped to the image coordinate system. Combined with the track segmentation results, the route map obtained based on the locomotive scheduling plan and actual scene mapping is matched with the track segmentation results to determine the travel track. Whether there is an obstacle on the track is determined based on the position of the unknown obstacle mapped to the image coordinate system, the position relationship between the position of the known obstacle and the track, and the sizes of the unknown obstacle and the known obstacle.

2. The method for detecting and locating obstacles in an unmanned railway system according to claim 1, wherein: The step of retaining the point cloud data of unknown obstacles in the point cloud data and clustering them, extracting features from the clustering results, and obtaining the position and size of the unknown obstacles specifically includes: Remove noise points and ground point clouds from the point cloud data collected by the lidar, retain the point cloud data representing obstacles, traverse each point in the point cloud data, take a point p as the starting point, find all points whose Euclidean distance to point p is less than a threshold and classify them into the same temporary cluster C; then repeat the Euclidean distance-based clustering process for each point in the temporary cluster C, continuously expanding the range of the temporary cluster C until no new points are added to the temporary cluster C; each temporary cluster C represents a potential unknown obstacle, and the center position of the temporary cluster C is used as the position of the unknown obstacle, and the maximum difference between the points of the temporary cluster C on each coordinate axis is used as the size of the unknown obstacle.

3. The method for detecting and locating obstacles in an unmanned railway system according to claim 1, wherein: The track image is input into a deep convolutional network to extract multi-scale features of the track image and segment the track line, specifically including: The deep convolutional network adopts the YOLOv8-seg network, segments the track lines in the track image through mask branches, and uses morphological operations to denoise the track line segmentation results.

4. A railway unmanned obstacle detection and positioning device, characterized in that: include: The calibration module jointly calibrates the lidar and camera so that the point cloud data collected by the lidar and the track image collected by the camera correspond to each other in the same coordinate system; The unknown obstacle detection module retains the point cloud data of unknown obstacles and clusters them. It then performs feature extraction on the clustering results to obtain the location and size of the unknown obstacles. Each temporary cluster C obtained by clustering represents a potential unknown obstacle. The center position of the temporary cluster C is used as the location of the unknown obstacle, and the maximum difference between the points in the temporary cluster C on each coordinate axis is used as the size of the unknown obstacle. The known obstacle detection module voxelizes the point cloud data of known obstacles using the PointPillars network to obtain voxelized cylinders. The convolutional neural network is used to extract the multi-scale features of the known obstacles in the voxelized cylinders, and the multi-scale features of the known obstacles are aggregated into single-scale features to obtain the location and size of the known obstacles. Track line segmentation module: inputs the track image into the deep convolutional network, extracts the multi-scale features of the track image, and realizes track line segmentation; The track matching module maps the positions of unknown obstacles and known obstacles to the image coordinate system. Combined with the track segmentation results, it matches the route map obtained based on the locomotive scheduling plan and actual scene mapping with the track segmentation results to determine the track. The track obstacle detection module determines whether there is an obstacle on the track based on the position of the unknown obstacle mapped to the image coordinate system, the position relationship between the position of the known obstacle and the track, and the sizes of the unknown obstacle and the known obstacle.

5. The railway unmanned obstacle detection and positioning device according to claim 4, characterized in that: In the unknown obstacle detection module, the point cloud data of unknown obstacles in the point cloud data is retained and clustered, and feature extraction is performed on the clustering results to obtain the position and size of the unknown obstacles, specifically including: Remove noise points and ground point clouds from the point cloud data collected by the lidar, retain the point cloud data representing obstacles, traverse each point in the point cloud data, take a point p as the starting point, find all points whose Euclidean distance to point p is less than a threshold and classify them into the same temporary cluster C; then repeat the Euclidean distance-based clustering process for each point in the temporary cluster C, continuously expanding the range of the temporary cluster C until no new points are added to the temporary cluster C; each temporary cluster C represents a potential unknown obstacle, and the center position of the temporary cluster C is used as the position of the unknown obstacle, and the maximum difference between the points of the temporary cluster C on each coordinate axis is used as the size of the unknown obstacle.

6. The railway unmanned obstacle detection and positioning device according to claim 4, characterized in that: In the track line segmentation module, the track image is input into the deep convolutional network to extract the multi-scale features of the track image to achieve track line segmentation, which specifically includes: The deep convolutional network adopts the YOLOv8-seg network, segments the track lines in the track image through mask branches, and uses morphological operations to denoise the track line segmentation results.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 3 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 3 are implemented.

Images