Railway unmanned obstacle detecting and positioning method, device, equipment and medium

Abstract

The invention relates to the technical field of auxiliary driving, and discloses a railway unmanned obstacle detection and positioning method, device and equipment and a medium, and the method comprises the steps: carrying out the clustering of the point cloud data of an unknown obstacle, carrying out the feature extraction of a clustering result, and obtaining the position and size of the unknown obstacle; voxelization is carried out on the point cloud data of the known obstacle, multi-scale features of the known obstacle in a voxelization cylinder are extracted through a convolutional neural network, and the position and the size of the known obstacle are obtained; inputting the track image into a deep convolutional network, extracting the multi-scale features of the track image, and achieving the segmentation of a track line; and judging an advancing track, and judging whether an obstacle exists on the advancing track or not according to the position relation between the positions of the unknown obstacle and the known obstacle and the advancing track and the size of the obstacle. The method is suitable for rail transportation in specific scenes such as railways and mines, the safety in the transportation process is guaranteed, and the transportation efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of assisted driving, and in particular to a method, device, equipment and medium for detecting and positioning obstacles in an unmanned railway driving system. Background Art

[0002] The research, development and operation of unmanned locomotive systems can help reduce the probability of transportation accidents caused by scheduling and operational errors; in addition, the unmanned vehicle system can operate 24 hours a day, which can improve transportation efficiency while reducing labor costs.

[0003] An important prerequisite for the safe operation of unmanned locomotive systems is that the locomotive can autonomously identify various obstacles in front of it, including pedestrians, cars, and obstacles of unknown categories. The method of identifying obstacles in the existing technology is to use a model trained based on scene data to identify obstacles. Its advantage is that it can learn the characteristics of various common obstacles and give accurate obstacle category information during model reasoning. However, unlike rail transit, industrial transportation railways are not closed and their surrounding environment is more complex. If a model-based obstacle detection method is used, it is easy to miss uncommon obstacles.

[0004] Image-based obstacle detection has the advantages of low cost and rich semantic information. However, images are easily affected by lighting conditions and bad weather, have poor stability, and cannot provide accurate obstacle distance information, which poses a great challenge to the camera's single-mode recognition.

[0005] The obstacle detection method based on lidar can accurately measure the distance from the obstacle to the locomotive, but the radar point cloud data has fewer features and the data annotation cost is high.

[0006] The combination of cameras and lidar can combine the advantages of both to obtain accurate obstacle status information, including obstacle coordinates, the distance of the obstacle from the locomotive, obstacle category information and track area, but it is still difficult to distinguish between traveling tracks and non-traveling tracks in the switch area.

[0007] The route map is the locomotive route obtained based on the locomotive dispatch plan combined with actual scene mapping. The route is obtained by combining the locomotive positioning information and matched with the track area to distinguish between traveling tracks and non-traveling tracks. Summary of the invention

[0008] The present invention provides a method, device, equipment and medium for detecting and locating obstacles in an unmanned railway. Combined with a route map, the advantages of cameras and laser radars can be well utilized to solve the above-mentioned problems.

[0009] In order to solve the above technical problems, the present invention adopts the following technical solutions: In a first aspect, the present invention provides a method for detecting and locating obstacles in an unmanned railway driving system, comprising: Jointly calibrate the lidar and camera so that the point cloud data collected by the lidar and the track image collected by the camera correspond to each other in the same coordinate system; The point cloud data of unknown obstacles in the point cloud data are retained and clustered, and features are extracted from the clustering results to obtain the position and size of the unknown obstacles; The point cloud data of known obstacles are voxelized through the PointPillars network to obtain voxelized cylinders. The multi-scale features of the known obstacles in the voxelized cylinders are extracted through a convolutional neural network. The multi-scale features of the known obstacles are aggregated into single-scale features to obtain the position and size of the known obstacles. Input the track image into the deep convolutional network to extract the multi-scale features of the track image and achieve track line segmentation; The positions of unknown obstacles and known obstacles are mapped to the image coordinate system. Combined with the track segmentation results, the route map is matched with the track segmentation results to determine the travel track. Whether there is an obstacle on the travel track is determined based on the position of the unknown obstacle mapped to the image coordinate system, the position relationship between the position of the known obstacle and the travel track, and the sizes of the unknown obstacle and the known obstacle.

[0010] In one embodiment, the step of retaining the point cloud data of the unknown obstacles in the point cloud data and clustering them, extracting features from the clustering results, and obtaining the position and size of the unknown obstacles specifically includes: Remove noise points and ground point clouds from the point cloud data collected by the lidar, retain the point cloud data representing obstacles, traverse each point in the point cloud data, take a point p as the starting point, find all points whose Euclidean distance to point p is less than the threshold and classify them into the same temporary cluster C; then repeat the clustering process based on Euclidean distance for each point in the temporary cluster C, and continuously expand the range of the temporary cluster C until no new points are added to the temporary cluster C; each temporary cluster C represents a potential unknown obstacle, and the center position of the temporary cluster C is taken as the position of the unknown obstacle, and the maximum difference between the points of the temporary cluster C on each coordinate axis is taken as the size of the unknown obstacle.

[0011] In one embodiment, the step of inputting the track image into a deep convolutional network to extract multi-scale features of the track image and implement track line segmentation specifically includes: The deep convolutional network adopts the YOLOv8-seg network, segments the track lines in the track image through mask branches, and denoises the track line segmentation results using morphological operations.

[0012] In a second aspect, the present invention provides a railway unmanned obstacle detection and positioning device, comprising: The calibration module jointly calibrates the lidar and the camera so that the point cloud data collected by the lidar and the track image collected by the camera correspond to each other in the same coordinate system; The unknown obstacle detection module retains the point cloud data of unknown obstacles in the point cloud data and clusters them, extracts features from the clustering results, and obtains the position and size of the unknown obstacles; The known obstacle detection module voxelizes the point cloud data of known obstacles through the PointPillars network to obtain voxelized cylinders, extracts the multi-scale features of the known obstacles in the voxelized cylinders through the convolutional neural network, aggregates the multi-scale features of the known obstacles into single-scale features, and obtains the position and size of the known obstacles; Track line segmentation module: inputs the track image into the deep convolutional network, extracts the multi-scale features of the track image, and realizes the segmentation of the track line; The travel track matching module maps the positions of unknown obstacles and known obstacles to the image coordinate system, combines the track line segmentation results, matches the travel route map with the track line segmentation results, and determines the travel track; The track obstacle detection module determines whether there is an obstacle on the track according to the position of the unknown obstacle mapped to the image coordinate system, the position relationship between the position of the known obstacle and the track, and the sizes of the unknown obstacle and the known obstacle.

[0013] In one embodiment, in the unknown obstacle detection module, the point cloud data of the unknown obstacles in the point cloud data are retained and clustered, and feature extraction is performed on the clustering results to obtain the position and size of the unknown obstacles, specifically including: Remove noise points and ground point clouds from the point cloud data collected by the lidar, retain the point cloud data representing obstacles, traverse each point in the point cloud data, take a point p as the starting point, find all points whose Euclidean distance to point p is less than the threshold and classify them into the same temporary cluster C; then repeat the clustering process based on Euclidean distance for each point in the temporary cluster C, and continuously expand the range of the temporary cluster C until no new points are added to the temporary cluster C; each temporary cluster C represents a potential unknown obstacle, and the center position of the temporary cluster C is taken as the position of the unknown obstacle, and the maximum difference between the points of the temporary cluster C on each coordinate axis is taken as the size of the unknown obstacle.

[0014] In one embodiment, in the track line segmentation module, the track image is input into a deep convolutional network to extract multi-scale features of the track image to achieve track line segmentation, specifically including: The deep convolutional network adopts the YOLOv8-seg network, segments the track lines in the track image through mask branches, and denoises the track line segmentation results using morphological operations.

[0015] In a third aspect, the present invention provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and wherein the processor implements the steps of the method of any one of the embodiments in the first aspect when executing the computer program.

[0016] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, characterized in that when the computer program is executed by a processor, the steps of the method of any one of the embodiments in the first aspect are implemented.

[0017] Compared with the prior art, the beneficial technical effects of the present invention are: (1) The present invention can ensure the safe operation of vehicles or equipment in complex environments, especially in scenarios where unknown obstacles exist and track lines need to be accurately identified.

[0018] (2) The present invention is applicable to rail transportation in specific scenarios such as railways and mines, so that the safety of the transportation process is guaranteed, the transportation efficiency is improved, and manual intervention is reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 4 is a flow chart of a method in an embodiment of the present invention.

[0020] Figure 2 Schematic diagram of a system used in an embodiment of the present invention.

[0021] Figure 3 It is a schematic diagram of an obstacle on a non-travel track when a locomotive is operating normally in an embodiment of the present invention.

[0022] Figure 4 It is a schematic diagram of an obstacle on the traveling track when the locomotive is operating normally in an embodiment of the present invention.

[0023] Figure 5 It is a schematic diagram of an embodiment of the present invention in which an obstacle is not on the track during normal operation of the locomotive. DETAILED DESCRIPTION

[0024] A preferred embodiment of the present invention is described in detail below with reference to the accompanying drawings.

[0025] like Figure 2As shown, a method for detecting and locating obstacles in an unmanned railway in the present invention, the detection system used includes: a laser radar and a camera (which may include a near-focus camera and a far-focus camera) installed on a vehicle, connected to a processor via Ethernet, and the detection result is directly sent to a vehicle controller. The detection system of the present invention can be applied to locomotives, such as industrial railway locomotives, to realize obstacle detection in a switch area. Industrial railway locomotives can be divided into unmanned (driving) locomotives and manned (driving) locomotives.

[0026] The following will take the obstacle detection and positioning of the UAV when running in the switch area as an example to illustrate the technical solution of the present invention.

[0027] like Figure 1 As shown, a method for detecting and locating obstacles in an unmanned railway driving system in the present invention comprises the following steps: S1: Jointly calibrate the lidar and camera so that the point cloud data collected by the lidar and the track image collected by the camera correspond to each other in the same coordinate system; S2: retain the point cloud data of unknown obstacles in the point cloud data and perform clustering, perform feature extraction on the clustering results, and obtain the position and size of the unknown obstacles; S3: The point cloud data of the known obstacles are voxelized through the PointPillars network to obtain voxelized cylinders. The multi-scale features of the known obstacles in the voxelized cylinders are extracted through a convolutional neural network. The multi-scale features of the known obstacles are aggregated into single-scale features to obtain the position and size of the known obstacles. S4: Input the track image into the deep convolutional network to extract the multi-scale features of the track image and achieve track line segmentation; S5: Mapping the positions of unknown obstacles and known obstacles to the image coordinate system, combining the track segmentation results, matching the route map with the track segmentation results, and determining the travel track; S6: judging whether there is an obstacle on the traveling track according to the position of the unknown obstacle mapped to the image coordinate system, the position relationship between the position of the known obstacle and the traveling track, and the sizes of the unknown obstacle and the known obstacle.

[0028] Specifically, in step S1, the laser radar and the camera are jointly calibrated, which is the basis of the entire detection system and is related to the accuracy of subsequent multi-sensor data fusion. The laser radar can accurately measure the three-dimensional spatial position and distance information of the target object, while the camera can capture rich visual images, including the texture, color and other semantic features of the target. In order to make the data of the two accurately correspond in the same coordinate system, it is necessary to calibrate the external parameters of the laser radar and the internal parameters of the camera.

[0029] The external parameters of the LiDAR mainly include the rotation matrix and translation matrix relative to the camera and the carrying platform (such as a vehicle or robot). The rotation matrix describes the rotation relationship of the LiDAR coordinate system relative to other coordinate systems, and the translation matrix represents its position offset. By accurately calculating these two matrices, the point cloud data collected by the LiDAR can be accurately mapped to the image coordinate system of the camera.

[0030] The internal parameters of the camera, such as the coordinates of the camera's main light spot and the focal length, have an important impact on the imaging effect and accuracy of the image. The coordinates of the camera's main light spot determine the center position of the image, and the focal length determines the image scale. By calibrating these parameters, image distortion can be eliminated and the quality of image data can be improved. After calibration, the lidar uses the original point cloud data and the camera uses the original image data to retain the most complete information and provide a rich data foundation for subsequent processing.

[0031] In one embodiment, step S2 of retaining the point cloud data of unknown obstacles in the point cloud data and clustering them, performing feature extraction on the clustering results, and obtaining the position and size of the unknown obstacles specifically includes: Remove noise points and ground point clouds from the point cloud data collected by the lidar, retain the point cloud data representing obstacles, traverse each point in the point cloud data, take a point p as the starting point, find all points whose Euclidean distance to point p is less than the threshold and classify them into the same temporary cluster C; then repeat the clustering process based on Euclidean distance for each point in the temporary cluster C, and continuously expand the range of the temporary cluster C until no new points are added to the temporary cluster C; each temporary cluster C represents a potential unknown obstacle, and the center position of the temporary cluster C is taken as the position of the unknown obstacle, and the maximum difference between the points of the temporary cluster C on each coordinate axis is taken as the size of the unknown obstacle.

[0032] Specifically, the present invention uses the Euclidean clustering algorithm to obtain unknown obstacle information in the following manner.

[0033] 1) After using LiDAR to collect point cloud data, since there is a lot of noise in the original point cloud data, such as ground point clouds, stray reflection points, etc., it will interfere with the subsequent obstacle detection. Therefore, it is necessary to pre-process the point cloud data collected by the LiDAR, and use statistical filtering, radius filtering and other methods to filter out noise points, remove irrelevant information such as noise points and ground point clouds, and retain point cloud data that may represent obstacles. The Euclidean clustering algorithm performs clustering based on the Euclidean distance between points. The Euclidean distance refers to the straight-line distance between two points in Euclidean space. For any two points in the point cloud data, by calculating the Euclidean distance between them, when the distance is less than the set threshold, the two points are considered to belong to the same cluster: For any two points in the point cloud and , the Euclidean distance between them The calculation formula is: ; When the Euclidean distance between two points is less than the set threshold , the two points are considered to belong to the same cluster.

[0034] 2) Use the Euclidean clustering algorithm to divide the processed point cloud data into different clusters. The algorithm will traverse each point in the point cloud, starting from a point p, and find all points whose Euclidean distance to it is less than the threshold. points, and classify them into the same temporary cluster C. Then repeat the above process for each point in the temporary cluster C, and continue to expand the range of the cluster until no new points can be added to the cluster. Each final cluster represents a potential unknown obstacle.

[0035] 3) Extract features from each cluster, including calculating the cluster’s center position, size, point cloud density and other features, so as to describe and analyze unknown obstacles later. , Cluster The i-th point in cluster Central location The calculation formula is: ; ; ; yes The x-axis coordinate, y-axis coordinate, and z-axis coordinate of the cluster; the size of the cluster can be determined by calculating the maximum difference between the points in the cluster on each coordinate axis, and the point cloud density can be calculated by the ratio of the number of points n in the cluster to the spatial volume V occupied by the cluster.

[0036] Specifically, the clustering process of point cloud data includes: Establishing a search structure: In order to improve search efficiency, a data structure called KD-Tree (K-Dimensional Tree, KD-Tree) is usually used to organize point cloud data. KD-Tree is a binary search tree that recursively divides the point cloud space into different regions, so that the search range can be quickly narrowed when looking for the neighborhood points of a certain point.

[0037] Select starting point: Randomly select an unvisited point from the point cloud as the starting point.

[0038] Neighborhood search: With the starting point as the center, search the KD tree for all nodes whose Euclidean distance to it is less than the set threshold. These points constitute the neighborhood of the starting point. The choice of is very critical, it determines the tightness of the clustering. If it is too large, different obstacles may be merged into one cluster. If it is too small, one obstacle may be split into multiple clusters.

[0039] Expanding the cluster: Mark the starting point and the points in its neighborhood as belonging to the same temporary cluster C, and repeat the above neighborhood search and marking process for each point in the neighborhood, continuously expanding the range of the cluster until no new points can be added to the cluster.

[0040] Repeat the above steps: Continue to select unvisited points as new starting points, and repeat the above process until all points in the point cloud have been visited. Eventually, each formed cluster represents a potential unknown obstacle.

[0041] In one embodiment, in step S3, the known obstacles are detected by the following steps: S31, extracting key point cloud data containing obstacles from the point cloud data, and screening and intercepting the key point cloud data as sample point clouds for establishing a data set for obstacle detection.

[0042] S32, perform data enhancement processing on the acquired sample point cloud, perform data annotation on the obstacle point cloud, including feature information such as obstacle category, 3D target box, direction, etc., store the obstacle point cloud and annotation information as a data set in KITTI format, and divide the obtained data set into a training set and a test set.

[0043] S33, construct an obstacle detection model, and formulate a data configuration file and a model configuration file of the obstacle detection model according to the analysis results of the collected point cloud data. The obstacle detection model is based on a feature extraction network constructed by a PointPillars network. As a three-dimensional target detection model, the PointPillars network can process three-dimensional point clouds into two-dimensional pseudo images, and then use traditional convolutional neural networks to extract features from the pseudo images, greatly improving the processing speed of point cloud data.

[0044] S34, training an obstacle detection model. After the training is completed, the obstacle point cloud to be detected is input into the trained obstacle detection model for detection, thereby completing the detection and identification of known obstacles in the industrial railway scenario.

[0045] The obstacle point cloud to be detected is input into the trained obstacle detection model for detection, and the detection and identification of known obstacles in the industrial railway scenario are completed. Specifically, the following steps are included: S341, first voxelization is performed to divide the three-dimensional point cloud data into vertical voxelized pillars (Pillars), each of which contains a certain number of points. The purpose of this is to convert the three-dimensional point cloud data into a two-dimensional representation for subsequent processing.

[0046] S342, then use the PointPillars network to perform feature extraction and obstacle detection. The structure of the PointPillars network mainly includes three parts: voxelized cylinder feature encoding, backbone network and detection head.

[0047] S343, voxelized cylinder feature encoding: For each point in the voxelized cylinder, calculate its offset relative to the center of the cylinder, its relative position in the voxelized cylinder and other features, and combine these features with the original coordinates and reflection intensity of the point to obtain the enhanced features of each point. Then, perform a maximum pooling operation on the point features in each voxelized cylinder to obtain the feature representation of each voxelized cylinder.

[0048] S344, Backbone network: Use a two-dimensional convolutional neural network to process the voxelized pillars. Usually a structure similar to the residual network (ResNet) is adopted, which contains multiple convolutional layers and residual blocks. These convolutional layers and residual blocks can learn different features of point cloud data, such as the shape and edges of objects. Through convolution operations of different scales, the network can extract multi-scale features. For example, a small-scale convolution kernel can capture the detailed information of obstacles, while a large-scale convolution kernel can grasp the overall distribution of obstacles. Finally, the extracted multi-scale features are aggregated and converted into single-scale features for subsequent accurate detection and positioning of known obstacles.

[0049] S345: Detection Head: Based on the extracted single-scale features, the detection head is responsible for predicting the 3D position, size, direction, and category of known obstacles in space. An anchor-based detection method is usually used to predefine anchors of different scales and proportions, and determine the obstacle information corresponding to each anchor through regression and classification operations.

[0050] In one embodiment, in step S4, the track image is input into a deep convolutional network to extract multi-scale features of the track image to achieve track line segmentation, which specifically includes: The deep convolutional network adopts the YOLOv8-seg network, segments the track lines in the track image through mask branches, and denoises the track line segmentation results using morphological operations.

[0051] The technical solution of step S4 is introduced in detail below.

[0052] Track images are collected through the camera and input into the deep convolutional network based on the YOLOv8-seg network for processing. The YOLOv8-seg network is improved on the basis of the YOLOv8 network by adding a mask branch to the detection head. It is a single-stage image segmentation algorithm. In the training phase, the YOLOv8-seg network optimizes the detection and segmentation tasks at the same time, and ensures that both tasks can be improved through the joint loss function; in the inference phase, the YOLOv8-seg network can output the bounding box of the target and the pixel-level mask of the corresponding area, thereby realizing instant target detection and segmentation. The overall architecture of the YOLOv8-seg network consists of three parts: the backbone network (Backbone), the neck network (Neck) and the detection head (Head).

[0053] Backbone network: used to extract features from track images. The feature extraction process is usually accompanied by a decrease in feature map resolution and an increase in the number of channels. The track image resolution is gradually downsampled from 640x640 to 20x20, and the number of input channels is increased from 3 to 512. The backbone network is mainly composed of CBS modules, C2f modules, and SPPF modules.

[0054] The CBS module represents a convolution group consisting of a convolution layer (Conv), a batch normalization layer (BN), and a SILU function. There are a total of 5 CBS modules in the backbone network, so 5 downsamplings will be performed. The C2f module is designed to deeply extract image features. The size of its output feature map and input feature map remains unchanged, but there are multiple branches inside it. It divides the feature map in the channel dimension. One part of the feature map is extracted through the convolution layer, and the other part of the feature map directly skips the convolution layer. Finally, the two parts of the feature map are spliced. This structure can reduce the amount of network calculation and enhance the expression ability of the features; the SPPF module consists of pooling operations of different scales, which splices feature maps of different scales together to improve the detection ability of targets of different sizes. The backbone network extracts multi-scale features of the image through convolution layers and pooling layers of different scales, providing rich information for subsequent track line segmentation.

[0055] Neck network: Using the Path Aggregation Network (PANet) structure, features are fused through bottom-up and top-down paths. The bottom-up path can transmit low-level feature information, which contains rich detail information; the top-down path can transmit high-level feature information, which has strong semantic information. In this way, feature information of different scales can complement each other, improve the expressiveness of features, and enhance the recognition ability of track lines.

[0056] Detection head: The decoupled head design is used to process the classification and regression tasks separately, and the detection and segmentation branches are combined. The classification task is responsible for determining whether each area in the image belongs to the track line, and the regression task is responsible for predicting the position and shape of the track line. This decoupled design can reduce the interference between tasks and improve the accuracy of detection. In the track line segmentation task, the feature map output by the neck network is combined with the mask branch to predict and segment the track line in the image. By analyzing the image features, the position and shape of the track line in the image are identified, and the track line segmentation result is generated.

[0057] Finally, the segmentation results are post-processed, such as using morphological operations (dilation, erosion, etc.) to remove noise and mis-segmented parts, and using connected region analysis to merge or remove small segmented areas to improve the accuracy and reliability of track segmentation.

[0058] In one embodiment, the positions of unknown obstacles and known obstacles are mapped to the image coordinate system in S5, and the route map is matched with the track segmentation result in combination with the track segmentation result to determine the travel track, which specifically includes: The positions of the unknown obstacles detected in step S2 and the known obstacles detected in step S3 are mapped from the LiDAR coordinate system to the image coordinate system. This requires using the rotation matrix and translation matrix calibrated in step 1 to convert the 3D point cloud coordinates in the LiDAR coordinate system into 2D pixel coordinates in the image coordinate system through the coordinate transformation formula.

[0059] After ensuring that the obstacle information and image information are in the same coordinate system, the route map is matched with the track segmentation result in combination with the track segmentation result in the image. Specifically, a feature matching algorithm can be used, such as a key point matching-based method. First, feature points such as corner points and edge points are extracted from the route map and the track segmentation result respectively. Then, descriptors (such as SIFT, SURF, etc.) are used to describe the features of these feature points. Finally, by comparing the descriptors of the feature points, the corresponding feature points in the route map and the track segmentation result are found, thereby determining the position of the track.

[0060] In one embodiment, step S6 determines whether there is an obstacle on the travel track according to the position of the unknown obstacle mapped to the image coordinate system, the position relationship between the position of the known obstacle and the travel track, and the sizes of the unknown obstacle and the known obstacle, and specifically includes: Analyze the relationship between the position of the obstacle mapped to the image and the travel track to determine whether there is an obstacle on the travel track.

[0061] It can also include: if there is an obstacle, further determine the type (unknown or known), location and size of the obstacle. According to the situation of the obstacle, provide a basis for subsequent decision-making and control.

[0062] For example, if the obstacle is close to the locomotive and is on the track, braking or avoidance measures may be required. The relative distance and relative speed between the obstacle and the locomotive can be calculated to predict the collision time, and the measures to be taken can be determined based on the length of the collision time. If the collision time is short, braking and stopping are required immediately; if the collision time is long, avoidance measures can be taken, such as honking the horn to slow down.

[0063] If the obstacle is far away or not on the track, you can continue to drive normally, but you need to be vigilant. You can set a certain safety distance threshold and warning area. When the obstacle enters the warning area, a warning signal is issued to remind the driver or control system to pay attention to the obstacle.

[0064] As an example, Figure 3 , Figure 4 and Figure 5 The following are schematic diagrams of the situation when the locomotive is operating normally and the obstacle ahead is on the non-travel track, on the travel track, and not on the track. For the first case, if Figure 3 As shown in the figure, track A is a non-travel track and track B is a travel track. At this time, the obstacle is on the non-travel track and does not affect the driving safety of the locomotive. In this case, braking is not required, but early warning is required. For the second case, Figure 4 As shown in the figure, the obstacle is on the track and the vehicle needs to be safely braked to stop. Figure 5 As shown, the obstacle is not on the track at this time and does not affect the driving safety of the locomotive. In this case, braking is not required, but driving warning is required.

[0065] In summary, the present invention provides a method for detecting and locating obstacles on unmanned railways, which makes a fusion judgment based on the point cloud data and track image data of the laser radar and combined with the travel route map to control the running state of the locomotive. The present invention can be applied to unmanned obstacle detection of locomotives on open industrial railways, so that the driving safety of locomotives during transportation can be guaranteed, and the overall transportation efficiency is improved.

[0066] It should be understood that, although the steps in the flowcharts of the accompanying drawings of the specification are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts of the accompanying drawings of the specification may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0067] Based on the description of the above method embodiment, the present invention also provides a device. The device may be a system (including a distributed system), software (application), module, component, server, client, etc. that uses the method described in the embodiment of this specification and is combined with necessary implementation hardware. Based on the same innovative concept, the device in one or more embodiments provided by the embodiment of the present disclosure is as described in the following embodiments. Since the implementation scheme of the device to solve the problem is similar to the method, the implementation of the specific device in the embodiment of this specification can refer to the implementation of the aforementioned method, and the repetitions will not be repeated. As used below, the term "module" or "module" is a combination of software and / or hardware that can implement a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0068] A railway unmanned driving obstacle detection and positioning device, comprising: The calibration module jointly calibrates the lidar and the camera so that the point cloud data collected by the lidar and the track image collected by the camera correspond to each other in the same coordinate system; The unknown obstacle detection module retains the point cloud data of unknown obstacles in the point cloud data and clusters them, extracts features from the clustering results, and obtains the position and size of the unknown obstacles; The known obstacle detection module voxelizes the point cloud data of known obstacles through the PointPillars network to obtain voxelized cylinders, extracts the multi-scale features of the known obstacles in the voxelized cylinders through the convolutional neural network, aggregates the multi-scale features of the known obstacles into single-scale features, and obtains the position and size of the known obstacles; Track line segmentation module: inputs the track image into the deep convolutional network, extracts the multi-scale features of the track image, and realizes the segmentation of the track line; The travel track matching module maps the positions of unknown obstacles and known obstacles to the image coordinate system, combines the track line segmentation results, matches the travel route map with the track line segmentation results, and determines the travel track; The track obstacle detection module determines whether there is an obstacle on the track according to the position of the unknown obstacle mapped to the image coordinate system, the position relationship between the position of the known obstacle and the track, and the sizes of the unknown obstacle and the known obstacle.

[0069] In one embodiment, in the unknown obstacle detection module, the point cloud data of the unknown obstacles in the point cloud data are retained and clustered, and feature extraction is performed on the clustering results to obtain the position and size of the unknown obstacles, specifically including: Remove noise points and ground point clouds from the point cloud data collected by the lidar, retain the point cloud data representing obstacles, traverse each point in the point cloud data, take a point p as the starting point, find all points whose Euclidean distance to point p is less than the threshold and classify them into the same temporary cluster C; then repeat the clustering process based on Euclidean distance for each point in the temporary cluster C, and continuously expand the range of the temporary cluster C until no new points are added to the temporary cluster C; each temporary cluster C represents a potential unknown obstacle, and the center position of the temporary cluster C is taken as the position of the unknown obstacle, and the maximum difference between the points of the temporary cluster C on each coordinate axis is taken as the size of the unknown obstacle.

[0070] In one embodiment, in the track line segmentation module, the track image is input into a deep convolutional network to extract multi-scale features of the track image to achieve track line segmentation, specifically including: The deep convolutional network adopts the YOLOv8-seg network, segments the track lines in the track image through mask branches, and denoises the track line segmentation results using morphological operations.

[0071] In one embodiment, the present invention provides a computer device, which may be a server. The computer device includes a processor, a memory and a network interface connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data used in the above method. The network interface of the computer device is used to communicate with an external terminal via a network connection. The computer program is executed by the processor to implement the above method.

[0072] In one embodiment, the present invention further provides a computer-readable storage medium including instructions, such as a memory including instructions, and the instructions can be executed by a processor to perform the above method. The storage medium can be a computer-readable storage medium, for example, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0073] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0074] It is obvious to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting, and the scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention, and any reference numerals in the claims should not be regarded as limiting the claims involved.

Claims

1. A method for detecting and locating obstacles in an unmanned railway system, characterized in that: include: Jointly calibrate the lidar and camera so that the point cloud data collected by the lidar and the track image collected by the camera correspond to each other in the same coordinate system; The point cloud data of unknown obstacles in the point cloud data are retained and clustered, and features are extracted from the clustering results to obtain the position and size of the unknown obstacles; The point cloud data of known obstacles are voxelized through the PointPillars network to obtain voxelized cylinders. The multi-scale features of the known obstacles in the voxelized cylinders are extracted through a convolutional neural network. The multi-scale features of the known obstacles are aggregated into single-scale features to obtain the position and size of the known obstacles. Input the track image into the deep convolutional network to extract the multi-scale features of the track image and achieve track line segmentation; The positions of unknown obstacles and known obstacles are mapped to the image coordinate system. Combined with the track segmentation results, the route map is matched with the track segmentation results to determine the travel track. Whether there is an obstacle on the travel track is determined based on the position of the unknown obstacle mapped to the image coordinate system, the position relationship between the position of the known obstacle and the travel track, and the sizes of the unknown obstacle and the known obstacle.

2. A railway unmanned driving obstacle detection and positioning method according to claim 1, characterized in that: The step of retaining the point cloud data of the unknown obstacles in the point cloud data and clustering them, extracting features from the clustering results, and obtaining the position and size of the unknown obstacles specifically includes: Remove noise points and ground point clouds from the point cloud data collected by the lidar, retain the point cloud data representing obstacles, traverse each point in the point cloud data, take a point p as the starting point, find all points whose Euclidean distance to point p is less than the threshold and classify them into the same temporary cluster C; then repeat the clustering process based on Euclidean distance for each point in the temporary cluster C, and continuously expand the range of the temporary cluster C until no new points are added to the temporary cluster C; each temporary cluster C represents a potential unknown obstacle, and the center position of the temporary cluster C is taken as the position of the unknown obstacle, and the maximum difference between the points of the temporary cluster C on each coordinate axis is taken as the size of the unknown obstacle.

3. The method for detecting and locating obstacles in an unmanned railway driving system according to claim 1, characterized in that: The track image is input into a deep convolutional network to extract multi-scale features of the track image and segment the track line, specifically including: The deep convolutional network adopts the YOLOv8-seg network, segments the track lines in the track image through mask branches, and denoises the track line segmentation results using morphological operations.

4. A railway unmanned obstacle detection and positioning device, characterized in that: include: The calibration module jointly calibrates the lidar and the camera so that the point cloud data collected by the lidar and the track image collected by the camera correspond to each other in the same coordinate system; The unknown obstacle detection module retains the point cloud data of unknown obstacles in the point cloud data and clusters them, extracts features from the clustering results, and obtains the position and size of the unknown obstacles; The known obstacle detection module voxelizes the point cloud data of known obstacles through the PointPillars network to obtain voxelized cylinders, extracts the multi-scale features of the known obstacles in the voxelized cylinders through the convolutional neural network, aggregates the multi-scale features of the known obstacles into single-scale features, and obtains the position and size of the known obstacles; Track line segmentation module: inputs the track image into the deep convolutional network, extracts the multi-scale features of the track image, and realizes the segmentation of the track line; The travel track matching module maps the positions of unknown obstacles and known obstacles to the image coordinate system, combines the track line segmentation results, matches the travel route map with the track line segmentation results, and determines the travel track; The track obstacle detection module determines whether there is an obstacle on the track according to the position of the unknown obstacle mapped to the image coordinate system, the position relationship between the position of the known obstacle and the track, and the sizes of the unknown obstacle and the known obstacle.

5. The unmanned railway obstacle detection and positioning device according to claim 4, characterized in that: In the unknown obstacle detection module, the point cloud data of unknown obstacles in the point cloud data are retained and clustered, and feature extraction is performed on the clustering results to obtain the position and size of the unknown obstacles, which specifically includes: Remove noise points and ground point clouds from the point cloud data collected by the lidar, retain the point cloud data representing obstacles, traverse each point in the point cloud data, take a point p as the starting point, find all points whose Euclidean distance to point p is less than the threshold and classify them into the same temporary cluster C; then repeat the clustering process based on Euclidean distance for each point in the temporary cluster C, and continuously expand the range of the temporary cluster C until no new points are added to the temporary cluster C; each temporary cluster C represents a potential unknown obstacle, and the center position of the temporary cluster C is taken as the position of the unknown obstacle, and the maximum difference between the points of the temporary cluster C on each coordinate axis is taken as the size of the unknown obstacle.

6. The unmanned railway obstacle detection and positioning device according to claim 4, characterized in that: In the track line segmentation module, the track image is input into the deep convolutional network to extract the multi-scale features of the track image to achieve track line segmentation, which specifically includes: The deep convolutional network adopts the YOLOv8-seg network, segments the track lines in the track image through mask branches, and denoises the track line segmentation results using morphological operations.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 3 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 3 are implemented.

Images